2026-04-30 16:29:10,652 [ 150632 ] INFO : ClickHouse root is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse (runner:42, check_args_and_update_paths) 2026-04-30 16:29:10,653 [ 150632 ] INFO : Cases dir is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration (runner:86, check_args_and_update_paths) 2026-04-30 16:29:10,653 [ 150632 ] INFO : utils dir is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse/utils (runner:97, check_args_and_update_paths) 2026-04-30 16:29:10,653 [ 150632 ] INFO : base_configs_dir: /home/ubuntu/_work/ClickHouse/ClickHouse/programs/server, binary: /home/ubuntu/_work/_temp/test/build/clickhouse, cases_dir: /home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration (runner:99, check_args_and_update_paths) clickhouse_integration_tests_volume Running pytest container as: 'docker run --rm --name clickhouse_integration_tests_jvksbm --privileged --dns-search='.' --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-odbc-bridge:/clickhouse-odbc-bridge --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-library-bridge:/clickhouse-library-bridge --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/backupview:/ClickHouse/utils/backupview --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/grpc-client/pb2:/ClickHouse/utils/grpc-client/pb2 --volume=/run:/run/host:ro --volume=clickhouse_integration_tests_volume:/var/lib/docker -e DOCKER_DOTNET_CLIENT_TAG=11de0b29a15d -e DOCKER_HELPER_TAG=2cffe1eae894 -e DOCKER_BASE_TAG=1e0b53d756cf -e DOCKER_KERBEROS_KDC_TAG=9391ecdee8d7 -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=9bec2a638e6e -e DOCKER_MYSQL_JAVA_CLIENT_TAG=766bff31cfe4 -e DOCKER_MYSQL_JS_CLIENT_TAG=41ba7c2ec2a1 -e DOCKER_MYSQL_PHP_CLIENT_TAG=88be89c1e3b6 -e DOCKER_NGINX_DAV_TAG=b55ac9cd7519 -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=a4eff5c7f4d6 -e DOCKER_PYTHON_BOTTLE_TAG=caad4729259e -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600 -e CLICKHOUSE_USE_OLD_ANALYZER=1 -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS="--dist=loadfile -n 10 -rfEps --run-id=0 --color=no --durations=0 'test_distributed_format/test.py::test_single_file[test_cluster_internal_replication]' 'test_distributed_format/test.py::test_single_file[test_cluster_no_internal_replication]' 'test_distributed_format/test.py::test_single_file_old[test_cluster_internal_replication]' 'test_distributed_format/test.py::test_single_file_old[test_cluster_no_internal_replication]' 'test_distributed_format/test.py::test_two_files[test_cluster_internal_replication]' 'test_distributed_format/test.py::test_two_files[test_cluster_no_internal_replication]' test_distributed_insert_backward_compatibility/test.py::test_distributed_in_tuple test_distributed_inter_server_secret/test.py::test_insecure test_distributed_inter_server_secret/test.py::test_insecure_insert_async test_distributed_inter_server_secret/test.py::test_insecure_insert_sync 'test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_insecure_cluster[default-]' 'test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_insecure_cluster[nopass-]' 'test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_insecure_cluster[pass-foo]' 'test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_secure_cluster[default-]' 'test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_secure_cluster[nopass-]' 'test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_secure_cluster[pass-foo]' 'test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_insecure_cluster[default-]' 'test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_insecure_cluster[nopass-]' 'test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_insecure_cluster[pass-foo]' 'test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[default-]' 'test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[nopass-]' 'test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[pass-foo]' test_distributed_inter_server_secret/test.py::test_secure test_distributed_inter_server_secret/test.py::test_secure_cluster_distributed_over_distributed_different_users_cluster test_distributed_inter_server_secret/test.py::test_secure_cluster_distributed_over_distributed_different_users_remote test_distributed_inter_server_secret/test.py::test_secure_disagree test_distributed_inter_server_secret/test.py::test_secure_disagree_insert test_distributed_inter_server_secret/test.py::test_secure_insert_async test_distributed_inter_server_secret/test.py::test_secure_insert_buffer_async test_distributed_inter_server_secret/test.py::test_secure_insert_sync 'test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[default-]' 'test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[nopass-]' 'test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[pass-foo]' 'test_distributed_inter_server_secret/test.py::test_user_secure_cluster[default-]' 'test_distributed_inter_server_secret/test.py::test_user_secure_cluster[nopass-]' 'test_distributed_inter_server_secret/test.py::test_user_secure_cluster[pass-foo]' test_distributed_load_balancing/test.py::test_distributed_replica_max_ignored_errors test_distributed_load_balancing/test.py::test_load_balancing_default test_distributed_load_balancing/test.py::test_load_balancing_first_or_random test_distributed_load_balancing/test.py::test_load_balancing_hostname_levenshtein_distance test_distributed_load_balancing/test.py::test_load_balancing_in_order test_distributed_load_balancing/test.py::test_load_balancing_nearest_hostname 'test_distributed_load_balancing/test.py::test_load_balancing_priority_round_robin[dist_priority]' 'test_distributed_load_balancing/test.py::test_load_balancing_priority_round_robin[dist_priority_negative]' test_distributed_load_balancing/test.py::test_load_balancing_round_robin test_dotnet_client/test.py::test_dotnet_client test_encrypted_disk_replication/test.py::test_replication test_encrypted_disk_replication/test.py::test_zero_copy_replication test_endpoint_macro_substitution/test.py::test_different_types test_endpoint_macro_substitution/test.py::test_select_by_type test_executable_user_defined_functions_config_reload/test.py::test test_globs_in_filepath/test.py::test_deep_structure test_globs_in_filepath/test.py::test_linear_structure test_globs_in_filepath/test.py::test_strange_filenames test_globs_in_filepath/test.py::test_table_function_and_virtual_columns test_graphite_merge_tree/test.py::test_broken_partial_rollup test_graphite_merge_tree/test.py::test_combined_rules test_graphite_merge_tree/test.py::test_combined_rules_with_default test_graphite_merge_tree/test.py::test_multiple_output_blocks test_graphite_merge_tree/test.py::test_multiple_paths_and_versions test_graphite_merge_tree/test.py::test_path_dangling_pointer test_graphite_merge_tree/test.py::test_paths_not_matching_any_pattern test_graphite_merge_tree/test.py::test_rollup_aggregation test_graphite_merge_tree/test.py::test_rollup_aggregation_2 test_graphite_merge_tree/test.py::test_rollup_versions test_graphite_merge_tree/test.py::test_system_graphite_retentions test_graphite_merge_tree/test.py::test_wrong_rollup_config test_grpc_protocol_ssl/test.py::test_insecure_channel test_grpc_protocol_ssl/test.py::test_secure_channel test_grpc_protocol_ssl/test.py::test_wrong_client_certificate test_hedged_requests/test.py::test_async_connect test_hedged_requests/test.py::test_async_query_sending test_hedged_requests/test.py::test_combination1 test_hedged_requests/test.py::test_combination2 test_hedged_requests/test.py::test_combination3 test_hedged_requests/test.py::test_combination4 test_hedged_requests/test.py::test_initial_receive_timeout test_hedged_requests/test.py::test_long_query test_hedged_requests/test.py::test_receive_timeout1 test_hedged_requests/test.py::test_receive_timeout2 test_hedged_requests/test.py::test_send_data test_hedged_requests/test.py::test_send_data2 test_hedged_requests/test.py::test_send_table_status_sleep test_hedged_requests/test.py::test_send_table_status_sleep2 test_hedged_requests/test.py::test_stuck_replica test_hot_reload_storage_policy/test.py::test_hot_reload_policy test_hot_reload_storage_policy/test.py::test_hot_reload_policy_distributed_table test_http_handlers_config/test.py::test_absolute_path_static_handler test_http_handlers_config/test.py::test_config_static_handler test_http_handlers_config/test.py::test_defaults_http_handlers test_http_handlers_config/test.py::test_defaults_http_handlers_config_order test_http_handlers_config/test.py::test_dynamic_query_handler test_http_handlers_config/test.py::test_fixed_static_handler test_http_handlers_config/test.py::test_headers_in_response test_http_handlers_config/test.py::test_predefined_query_handler test_http_handlers_config/test.py::test_prometheus_handler test_http_handlers_config/test.py::test_relative_path_static_handler test_http_handlers_config/test.py::test_replicas_status_handler test_input_format_parallel_parsing_memory_tracking/test.py::test_memory_tracking_total 'test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_corrupted_big[0]' -vvv" altinityinfra/integration-tests-runner:37a9815fd2fa '. Start tests ============================= test session starts ============================== platform linux -- Python 3.10.12, pytest-7.4.4, pluggy-1.5.0 -- /usr/bin/python3 cachedir: .pytest_cache rootdir: /ClickHouse/tests/integration configfile: pytest.ini plugins: order-1.0.1, random-0.2, timeout-2.2.0, repeat-0.9.3, reportlog-0.4.0, xdist-3.5.0 timeout: 900.0s timeout method: signal timeout func_only: False created: 10/10 workers 10 workers [100 items] scheduling tests via LoadFileScheduling test_graphite_merge_tree/test.py::test_broken_partial_rollup test_distributed_inter_server_secret/test.py::test_insecure test_http_handlers_config/test.py::test_absolute_path_static_handler test_hedged_requests/test.py::test_async_connect test_distributed_format/test.py::test_single_file[test_cluster_internal_replication] test_globs_in_filepath/test.py::test_deep_structure test_grpc_protocol_ssl/test.py::test_insecure_channel test_endpoint_macro_substitution/test.py::test_different_types test_distributed_load_balancing/test.py::test_distributed_replica_max_ignored_errors test_encrypted_disk_replication/test.py::test_replication [gw7] [ 1%] PASSED test_grpc_protocol_ssl/test.py::test_insecure_channel test_grpc_protocol_ssl/test.py::test_secure_channel [gw7] [ 2%] PASSED test_grpc_protocol_ssl/test.py::test_secure_channel test_grpc_protocol_ssl/test.py::test_wrong_client_certificate [gw3] [ 3%] PASSED test_graphite_merge_tree/test.py::test_broken_partial_rollup [gw7] [ 4%] PASSED test_grpc_protocol_ssl/test.py::test_wrong_client_certificate test_graphite_merge_tree/test.py::test_combined_rules [gw5] [ 5%] PASSED test_distributed_format/test.py::test_single_file[test_cluster_internal_replication] test_distributed_format/test.py::test_single_file[test_cluster_no_internal_replication] [gw3] [ 6%] PASSED test_graphite_merge_tree/test.py::test_combined_rules test_graphite_merge_tree/test.py::test_combined_rules_with_default [gw5] [ 7%] PASSED test_distributed_format/test.py::test_single_file[test_cluster_no_internal_replication] test_distributed_format/test.py::test_single_file_old[test_cluster_internal_replication] [gw2] [ 8%] PASSED test_http_handlers_config/test.py::test_absolute_path_static_handler test_http_handlers_config/test.py::test_config_static_handler [gw3] [ 9%] PASSED test_graphite_merge_tree/test.py::test_combined_rules_with_default test_graphite_merge_tree/test.py::test_multiple_output_blocks test_dotnet_client/test.py::test_dotnet_client [gw5] [ 10%] PASSED test_distributed_format/test.py::test_single_file_old[test_cluster_internal_replication] test_distributed_format/test.py::test_single_file_old[test_cluster_no_internal_replication] [gw3] [ 11%] PASSED test_graphite_merge_tree/test.py::test_multiple_output_blocks test_graphite_merge_tree/test.py::test_multiple_paths_and_versions [gw3] [ 12%] PASSED test_graphite_merge_tree/test.py::test_multiple_paths_and_versions test_graphite_merge_tree/test.py::test_path_dangling_pointer [gw5] [ 13%] PASSED test_distributed_format/test.py::test_single_file_old[test_cluster_no_internal_replication] test_distributed_format/test.py::test_two_files[test_cluster_internal_replication] [gw1] [ 14%] PASSED test_distributed_inter_server_secret/test.py::test_insecure test_distributed_inter_server_secret/test.py::test_insecure_insert_async [gw5] [ 15%] PASSED test_distributed_format/test.py::test_two_files[test_cluster_internal_replication] test_distributed_format/test.py::test_two_files[test_cluster_no_internal_replication] [gw1] [ 16%] PASSED test_distributed_inter_server_secret/test.py::test_insecure_insert_async test_distributed_inter_server_secret/test.py::test_insecure_insert_sync [gw5] [ 17%] PASSED test_distributed_format/test.py::test_two_files[test_cluster_no_internal_replication] [gw1] [ 18%] PASSED test_distributed_inter_server_secret/test.py::test_insecure_insert_sync test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_insecure_cluster[default-] [gw4] [ 19%] FAILED test_distributed_load_balancing/test.py::test_distributed_replica_max_ignored_errors test_distributed_load_balancing/test.py::test_load_balancing_default [gw1] [ 20%] PASSED test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_insecure_cluster[default-] test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_insecure_cluster[nopass-] test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_corrupted_big[0] [gw1] [ 21%] PASSED test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_insecure_cluster[nopass-] test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_insecure_cluster[pass-foo] [gw1] [ 22%] PASSED test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_insecure_cluster[pass-foo] test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_secure_cluster[default-] [gw1] [ 23%] PASSED test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_secure_cluster[default-] test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_secure_cluster[nopass-] [gw0] [ 24%] PASSED test_hedged_requests/test.py::test_async_connect test_hedged_requests/test.py::test_async_query_sending [gw1] [ 25%] PASSED test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_secure_cluster[nopass-] test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_secure_cluster[pass-foo] [gw1] [ 26%] PASSED test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_secure_cluster[pass-foo] test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_insecure_cluster[default-] [gw1] [ 27%] PASSED test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_insecure_cluster[default-] test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_insecure_cluster[nopass-] [gw3] [ 28%] PASSED test_graphite_merge_tree/test.py::test_path_dangling_pointer test_graphite_merge_tree/test.py::test_paths_not_matching_any_pattern [gw3] [ 29%] PASSED test_graphite_merge_tree/test.py::test_paths_not_matching_any_pattern test_graphite_merge_tree/test.py::test_rollup_aggregation [gw1] [ 30%] PASSED test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_insecure_cluster[nopass-] test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_insecure_cluster[pass-foo] [gw8] [ 31%] PASSED test_encrypted_disk_replication/test.py::test_replication test_encrypted_disk_replication/test.py::test_zero_copy_replication [gw3] [ 32%] PASSED test_graphite_merge_tree/test.py::test_rollup_aggregation test_graphite_merge_tree/test.py::test_rollup_aggregation_2 [gw1] [ 33%] PASSED test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_insecure_cluster[pass-foo] test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[default-] [gw3] [ 34%] PASSED test_graphite_merge_tree/test.py::test_rollup_aggregation_2 test_graphite_merge_tree/test.py::test_rollup_versions [gw8] [ 35%] PASSED test_encrypted_disk_replication/test.py::test_zero_copy_replication [gw1] [ 36%] PASSED test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[default-] test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[nopass-] [gw2] [ 37%] PASSED test_http_handlers_config/test.py::test_config_static_handler test_http_handlers_config/test.py::test_defaults_http_handlers [gw3] [ 38%] PASSED test_graphite_merge_tree/test.py::test_rollup_versions test_graphite_merge_tree/test.py::test_system_graphite_retentions [gw1] [ 39%] PASSED test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[nopass-] test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[pass-foo] [gw7] [ 40%] PASSED test_dotnet_client/test.py::test_dotnet_client [gw3] [ 41%] PASSED test_graphite_merge_tree/test.py::test_system_graphite_retentions test_graphite_merge_tree/test.py::test_wrong_rollup_config [gw1] [ 42%] PASSED test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[pass-foo] test_distributed_inter_server_secret/test.py::test_secure [gw3] [ 43%] PASSED test_graphite_merge_tree/test.py::test_wrong_rollup_config [gw1] [ 44%] PASSED test_distributed_inter_server_secret/test.py::test_secure test_distributed_inter_server_secret/test.py::test_secure_cluster_distributed_over_distributed_different_users_cluster [gw1] [ 45%] PASSED test_distributed_inter_server_secret/test.py::test_secure_cluster_distributed_over_distributed_different_users_cluster test_distributed_inter_server_secret/test.py::test_secure_cluster_distributed_over_distributed_different_users_remote test_executable_user_defined_functions_config_reload/test.py::test [gw1] [ 46%] PASSED test_distributed_inter_server_secret/test.py::test_secure_cluster_distributed_over_distributed_different_users_remote test_distributed_inter_server_secret/test.py::test_secure_disagree [gw1] [ 47%] PASSED test_distributed_inter_server_secret/test.py::test_secure_disagree test_distributed_inter_server_secret/test.py::test_secure_disagree_insert [gw1] [ 48%] PASSED test_distributed_inter_server_secret/test.py::test_secure_disagree_insert test_distributed_inter_server_secret/test.py::test_secure_insert_async test_hot_reload_storage_policy/test.py::test_hot_reload_policy [gw9] [ 49%] PASSED test_endpoint_macro_substitution/test.py::test_different_types test_endpoint_macro_substitution/test.py::test_select_by_type [gw1] [ 50%] PASSED test_distributed_inter_server_secret/test.py::test_secure_insert_async test_distributed_inter_server_secret/test.py::test_secure_insert_buffer_async [gw9] [ 51%] PASSED test_endpoint_macro_substitution/test.py::test_select_by_type [gw0] [ 52%] PASSED test_hedged_requests/test.py::test_async_query_sending test_hedged_requests/test.py::test_combination1 [gw1] [ 53%] PASSED test_distributed_inter_server_secret/test.py::test_secure_insert_buffer_async test_distributed_inter_server_secret/test.py::test_secure_insert_sync [gw5] [ 54%] PASSED test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_corrupted_big[0] [gw1] [ 55%] PASSED test_distributed_inter_server_secret/test.py::test_secure_insert_sync test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[default-] [gw1] [ 56%] PASSED test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[default-] test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[nopass-] test_distributed_insert_backward_compatibility/test.py::test_distributed_in_tuple [gw1] [ 57%] PASSED test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[nopass-] test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[pass-foo] [gw2] [ 58%] PASSED test_http_handlers_config/test.py::test_defaults_http_handlers test_http_handlers_config/test.py::test_defaults_http_handlers_config_order [gw1] [ 59%] PASSED test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[pass-foo] test_distributed_inter_server_secret/test.py::test_user_secure_cluster[default-] [gw0] [ 60%] PASSED test_hedged_requests/test.py::test_combination1 test_hedged_requests/test.py::test_combination2 [gw1] [ 61%] PASSED test_distributed_inter_server_secret/test.py::test_user_secure_cluster[default-] test_distributed_inter_server_secret/test.py::test_user_secure_cluster[nopass-] [gw1] [ 62%] PASSED test_distributed_inter_server_secret/test.py::test_user_secure_cluster[nopass-] test_distributed_inter_server_secret/test.py::test_user_secure_cluster[pass-foo] [gw8] [ 63%] PASSED test_hot_reload_storage_policy/test.py::test_hot_reload_policy test_hot_reload_storage_policy/test.py::test_hot_reload_policy_distributed_table [gw1] [ 64%] PASSED test_distributed_inter_server_secret/test.py::test_user_secure_cluster[pass-foo] [gw7] [ 65%] PASSED test_executable_user_defined_functions_config_reload/test.py::test [gw0] [ 66%] PASSED test_hedged_requests/test.py::test_combination2 test_hedged_requests/test.py::test_combination3 test_input_format_parallel_parsing_memory_tracking/test.py::test_memory_tracking_total [gw8] [ 67%] PASSED test_hot_reload_storage_policy/test.py::test_hot_reload_policy_distributed_table [gw9] [ 68%] PASSED test_distributed_insert_backward_compatibility/test.py::test_distributed_in_tuple [gw0] [ 69%] PASSED test_hedged_requests/test.py::test_combination3 test_hedged_requests/test.py::test_combination4 [gw7] [ 70%] PASSED test_input_format_parallel_parsing_memory_tracking/test.py::test_memory_tracking_total [gw2] [ 71%] PASSED test_http_handlers_config/test.py::test_defaults_http_handlers_config_order test_http_handlers_config/test.py::test_dynamic_query_handler [gw0] [ 72%] PASSED test_hedged_requests/test.py::test_combination4 test_hedged_requests/test.py::test_initial_receive_timeout [gw4] [ 73%] PASSED test_distributed_load_balancing/test.py::test_load_balancing_default test_distributed_load_balancing/test.py::test_load_balancing_first_or_random [gw2] [ 74%] PASSED test_http_handlers_config/test.py::test_dynamic_query_handler test_http_handlers_config/test.py::test_fixed_static_handler [gw0] [ 75%] PASSED test_hedged_requests/test.py::test_initial_receive_timeout test_hedged_requests/test.py::test_long_query [gw2] [ 76%] PASSED test_http_handlers_config/test.py::test_fixed_static_handler test_http_handlers_config/test.py::test_headers_in_response [gw0] [ 77%] PASSED test_hedged_requests/test.py::test_long_query test_hedged_requests/test.py::test_receive_timeout1 [gw2] [ 78%] PASSED test_http_handlers_config/test.py::test_headers_in_response test_http_handlers_config/test.py::test_predefined_query_handler [gw6] [ 79%] PASSED test_globs_in_filepath/test.py::test_deep_structure test_globs_in_filepath/test.py::test_linear_structure [gw6] [ 80%] PASSED test_globs_in_filepath/test.py::test_linear_structure test_globs_in_filepath/test.py::test_strange_filenames [gw2] [ 81%] PASSED test_http_handlers_config/test.py::test_predefined_query_handler test_http_handlers_config/test.py::test_prometheus_handler [gw6] [ 82%] PASSED test_globs_in_filepath/test.py::test_strange_filenames test_globs_in_filepath/test.py::test_table_function_and_virtual_columns [gw6] [ 83%] PASSED test_globs_in_filepath/test.py::test_table_function_and_virtual_columns [gw2] [ 84%] PASSED test_http_handlers_config/test.py::test_prometheus_handler test_http_handlers_config/test.py::test_relative_path_static_handler [gw2] [ 85%] PASSED test_http_handlers_config/test.py::test_relative_path_static_handler test_http_handlers_config/test.py::test_replicas_status_handler [gw0] [ 86%] FAILED test_hedged_requests/test.py::test_receive_timeout1 test_hedged_requests/test.py::test_receive_timeout2 [gw2] [ 87%] PASSED test_http_handlers_config/test.py::test_replicas_status_handler [gw4] [ 88%] PASSED test_distributed_load_balancing/test.py::test_load_balancing_first_or_random test_distributed_load_balancing/test.py::test_load_balancing_hostname_levenshtein_distance [gw0] [ 89%] FAILED test_hedged_requests/test.py::test_receive_timeout2 test_hedged_requests/test.py::test_send_data [gw0] [ 90%] FAILED test_hedged_requests/test.py::test_send_data test_hedged_requests/test.py::test_send_data2 [gw0] [ 91%] FAILED test_hedged_requests/test.py::test_send_data2 test_hedged_requests/test.py::test_send_table_status_sleep [gw0] [ 92%] PASSED test_hedged_requests/test.py::test_send_table_status_sleep test_hedged_requests/test.py::test_send_table_status_sleep2 [gw4] [ 93%] PASSED test_distributed_load_balancing/test.py::test_load_balancing_hostname_levenshtein_distance test_distributed_load_balancing/test.py::test_load_balancing_in_order [gw0] [ 94%] PASSED test_hedged_requests/test.py::test_send_table_status_sleep2 test_hedged_requests/test.py::test_stuck_replica [gw0] [ 95%] PASSED test_hedged_requests/test.py::test_stuck_replica [gw4] [ 96%] PASSED test_distributed_load_balancing/test.py::test_load_balancing_in_order test_distributed_load_balancing/test.py::test_load_balancing_nearest_hostname [gw4] [ 97%] PASSED test_distributed_load_balancing/test.py::test_load_balancing_nearest_hostname test_distributed_load_balancing/test.py::test_load_balancing_priority_round_robin[dist_priority] [gw4] [ 98%] PASSED test_distributed_load_balancing/test.py::test_load_balancing_priority_round_robin[dist_priority] test_distributed_load_balancing/test.py::test_load_balancing_priority_round_robin[dist_priority_negative] [gw4] [ 99%] PASSED test_distributed_load_balancing/test.py::test_load_balancing_priority_round_robin[dist_priority_negative] test_distributed_load_balancing/test.py::test_load_balancing_round_robin [gw4] [100%] PASSED test_distributed_load_balancing/test.py::test_load_balancing_round_robin =================================== FAILURES =================================== _________________ test_distributed_replica_max_ignored_errors __________________ [gw4] linux -- Python 3.10.12 /usr/bin/python3 def test_distributed_replica_max_ignored_errors(): settings = { "use_hedged_requests": 0, "load_balancing": "in_order", "prefer_localhost_replica": 0, "connect_timeout": 2, "receive_timeout": 2, "send_timeout": 2, "idle_connection_timeout": 2, "tcp_keep_alive_timeout": 2, "distributed_replica_max_ignored_errors": 0, "distributed_replica_error_half_life": 60, } # initiate connection (if started only this test) n2.query("SELECT * FROM dist", settings=settings) cluster.pause_container("n1") # n1 paused -- skipping, and increment error_count for n1 # but the query succeeds, no need in query_and_get_error() n2.query("SELECT * FROM dist", settings=settings) # XXX: due to config reloading we need second time (sigh) n2.query("SELECT * FROM dist", settings=settings) # check error_count for n1 assert ( int( n2.query( """ SELECT errors_count FROM system.clusters WHERE cluster = 'replicas_cluster' AND host_name = 'n1' """, settings=settings, ) ) == 1 ) cluster.unpause_container("n1") # still n2 > assert get_node(n2, settings=settings) == "n2" E AssertionError: assert 'n3' == 'n2' E - n2 E + n3 test_distributed_load_balancing/test.py:234: AssertionError ---------------------------- Captured stdout setup ----------------------------- Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml ------------------------------ Captured log setup ------------------------------ 2026-04-30 16:29:21 [ 434 ] DEBUG : Command:['docker ps | wc -l'] (cluster.py:113, run_and_check) 2026-04-30 16:29:21 [ 434 ] DEBUG : Stdout:1 (cluster.py:121, run_and_check) 2026-04-30 16:29:21 [ 434 ] DEBUG : No running containers (conftest.py:95, cleanup_environment) 2026-04-30 16:29:21 [ 434 ] DEBUG : Pruning Docker networks (conftest.py:97, cleanup_environment) 2026-04-30 16:29:21 [ 434 ] DEBUG : Command:['docker network prune --force'] (cluster.py:113, run_and_check) 2026-04-30 16:29:21 [ 434 ] DEBUG : Command:["sysctl net.ipv4.ip_local_port_range='55000 65535'"] (cluster.py:113, run_and_check) 2026-04-30 16:29:21 [ 434 ] DEBUG : Stdout:net.ipv4.ip_local_port_range = 55000 65535 (cluster.py:121, run_and_check) 2026-04-30 16:29:21 [ 434 ] INFO : Running tests in /ClickHouse/tests/integration/test_distributed_load_balancing/test.py (cluster.py:2788, start) 2026-04-30 16:29:21 [ 434 ] DEBUG : Cluster start called. is_up=False (cluster.py:2795, start) 2026-04-30 16:29:21 [ 434 ] DEBUG : Docker networks for project roottestdistributedloadbalancing_gw4 are NETWORK ID NAME DRIVER SCOPE (cluster.py:855, print_all_docker_pieces) 2026-04-30 16:29:21 [ 434 ] DEBUG : Docker containers for project roottestdistributedloadbalancing_gw4 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:863, print_all_docker_pieces) 2026-04-30 16:29:21 [ 434 ] DEBUG : Docker volumes for project roottestdistributedloadbalancing_gw4 are DRIVER VOLUME NAME (cluster.py:871, print_all_docker_pieces) 2026-04-30 16:29:21 [ 434 ] DEBUG : Cleanup called (cluster.py:876, cleanup) 2026-04-30 16:29:22 [ 434 ] DEBUG : Docker networks for project roottestdistributedloadbalancing_gw4 are NETWORK ID NAME DRIVER SCOPE (cluster.py:855, print_all_docker_pieces) 2026-04-30 16:29:22 [ 434 ] DEBUG : Docker containers for project roottestdistributedloadbalancing_gw4 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:863, print_all_docker_pieces) 2026-04-30 16:29:22 [ 434 ] DEBUG : Docker volumes for project roottestdistributedloadbalancing_gw4 are DRIVER VOLUME NAME (cluster.py:871, print_all_docker_pieces) 2026-04-30 16:29:22 [ 434 ] DEBUG : Command:docker container list --all --filter name='^/roottestdistributedloadbalancing_gw4_.*_1$' --format '{{.ID}}:{{.Names}}' (cluster.py:113, run_and_check) 2026-04-30 16:29:22 [ 434 ] DEBUG : Unstopped containers: {} (cluster.py:890, cleanup) 2026-04-30 16:29:22 [ 434 ] DEBUG : No running containers for project: roottestdistributedloadbalancing_gw4 (cluster.py:904, cleanup) 2026-04-30 16:29:22 [ 434 ] DEBUG : Trying to prune unused networks... (cluster.py:910, cleanup) 2026-04-30 16:29:22 [ 434 ] DEBUG : Trying to prune unused images... (cluster.py:926, cleanup) 2026-04-30 16:29:22 [ 434 ] DEBUG : Command:['docker', 'image', 'prune', '-f'] (cluster.py:113, run_and_check) 2026-04-30 16:29:22 [ 434 ] DEBUG : Stderr:Error response from daemon: a prune operation is already running (cluster.py:123, run_and_check) 2026-04-30 16:29:22 [ 434 ] DEBUG : Exitcode:1 (cluster.py:125, run_and_check) 2026-04-30 16:29:22 [ 434 ] DEBUG : Trying to prune unused volumes... (cluster.py:935, cleanup) 2026-04-30 16:29:22 [ 434 ] DEBUG : Command:['docker volume ls | wc -l'] (cluster.py:113, run_and_check) 2026-04-30 16:29:22 [ 434 ] DEBUG : Stdout:1 (cluster.py:121, run_and_check) 2026-04-30 16:29:22 [ 434 ] DEBUG : Setup directory for instance: n1 (cluster.py:2808, start) 2026-04-30 16:29:22 [ 434 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4534, create_dir) 2026-04-30 16:29:22 [ 434 ] DEBUG : Create directory for common tests configuration (cluster.py:4539, create_dir) 2026-04-30 16:29:22 [ 434 ] DEBUG : Copy common configuration from helpers (cluster.py:4559, create_dir) 2026-04-30 16:29:22 [ 434 ] DEBUG : Generate and write macros file (cluster.py:4602, create_dir) 2026-04-30 16:29:22 [ 434 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_distributed_load_balancing/configs/remote_servers.xml'] to /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0_gw4/n1/configs/config.d (cluster.py:4632, create_dir) 2026-04-30 16:29:22 [ 434 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0_gw4/n1/database (cluster.py:4649, create_dir) 2026-04-30 16:29:22 [ 434 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0_gw4/n1/logs (cluster.py:4660, create_dir) 2026-04-30 16:29:22 [ 434 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log"] (cluster.py:4746, create_dir) 2026-04-30 16:29:22 [ 434 ] DEBUG : Setup directory for instance: n2 (cluster.py:2808, start) 2026-04-30 16:29:22 [ 434 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4534, create_dir) 2026-04-30 16:29:22 [ 434 ] DEBUG : Create directory for common tests configuration (cluster.py:4539, create_dir) 2026-04-30 16:29:22 [ 434 ] DEBUG : Copy common configuration from helpers (cluster.py:4559, create_dir) 2026-04-30 16:29:22 [ 434 ] DEBUG : Generate and write macros file (cluster.py:4602, create_dir) 2026-04-30 16:29:22 [ 434 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_distributed_load_balancing/configs/remote_servers.xml'] to /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0_gw4/n2/configs/config.d (cluster.py:4632, create_dir) 2026-04-30 16:29:22 [ 434 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0_gw4/n2/database (cluster.py:4649, create_dir) 2026-04-30 16:29:22 [ 434 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0_gw4/n2/logs (cluster.py:4660, create_dir) 2026-04-30 16:29:22 [ 434 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log"] (cluster.py:4746, create_dir) 2026-04-30 16:29:22 [ 434 ] DEBUG : Setup directory for instance: n3 (cluster.py:2808, start) 2026-04-30 16:29:22 [ 434 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4534, create_dir) 2026-04-30 16:29:22 [ 434 ] DEBUG : Create directory for common tests configuration (cluster.py:4539, create_dir) 2026-04-30 16:29:22 [ 434 ] DEBUG : Copy common configuration from helpers (cluster.py:4559, create_dir) 2026-04-30 16:29:22 [ 434 ] DEBUG : Generate and write macros file (cluster.py:4602, create_dir) 2026-04-30 16:29:22 [ 434 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_distributed_load_balancing/configs/remote_servers.xml'] to /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0_gw4/n3/configs/config.d (cluster.py:4632, create_dir) 2026-04-30 16:29:22 [ 434 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0_gw4/n3/database (cluster.py:4649, create_dir) 2026-04-30 16:29:22 [ 434 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0_gw4/n3/logs (cluster.py:4660, create_dir) 2026-04-30 16:29:22 [ 434 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log"] (cluster.py:4746, create_dir) 2026-04-30 16:29:22 [ 434 ] DEBUG : Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw'} stored in /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0_gw4/.env (cluster.py:86, _create_env_file) 2026-04-30 16:29:22 [ 434 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file) 2026-04-30 16:29:22 [ 434 ] DEBUG : No config file found (config.py:28, find_config_file) 2026-04-30 16:29:22 [ 434 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file) 2026-04-30 16:29:22 [ 434 ] DEBUG : No config file found (config.py:28, find_config_file) 2026-04-30 16:29:22 [ 434 ] DEBUG : http://localhost:None "GET /version HTTP/1.1" 200 824 (connectionpool.py:547, _make_request) 2026-04-30 16:29:22 [ 434 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0_gw4/.env', '--project-name', 'roottestdistributedloadbalancing_gw4', '--file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0_gw4/n1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0_gw4/n2/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0_gw4/n3/docker-compose.yml', 'pull'] (cluster.py:113, run_and_check) 2026-04-30 16:30:06 [ 434 ] DEBUG : Stderr:Pulling n3 ... (cluster.py:123, run_and_check) 2026-04-30 16:30:06 [ 434 ] DEBUG : Stderr:Pulling n1 ... (cluster.py:123, run_and_check) 2026-04-30 16:30:06 [ 434 ] DEBUG : Stderr:Pulling n2 ... (cluster.py:123, run_and_check) 2026-04-30 16:30:06 [ 434 ] DEBUG : Stderr:Pulling n1 ... pulling from altinityinfra/integr... (cluster.py:123, run_and_check) 2026-04-30 16:30:06 [ 434 ] DEBUG : Stderr:Pulling n2 ... pulling from altinityinfra/integr... (cluster.py:123, run_and_check) 2026-04-30 16:30:06 [ 434 ] DEBUG : Stderr:Pulling n3 ... pulling from altinityinfra/integr... (cluster.py:123, run_and_check) 2026-04-30 16:30:06 [ 434 ] DEBUG : Stderr:Pulling n1 ... digest: sha256:bf725030a292d5daab... (cluster.py:123, run_and_check) 2026-04-30 16:30:06 [ 434 ] DEBUG : Stderr:Pulling n1 ... status: image is up to date for a... (cluster.py:123, run_and_check) 2026-04-30 16:30:06 [ 434 ] DEBUG : Stderr:Pulling n3 ... digest: sha256:bf725030a292d5daab... (cluster.py:123, run_and_check) 2026-04-30 16:30:06 [ 434 ] DEBUG : Stderr:Pulling n3 ... status: image is up to date for a... (cluster.py:123, run_and_check) 2026-04-30 16:30:06 [ 434 ] DEBUG : Stderr:Pulling n1 ... done (cluster.py:123, run_and_check) 2026-04-30 16:30:06 [ 434 ] DEBUG : Stderr:Pulling n2 ... digest: sha256:bf725030a292d5daab... (cluster.py:123, run_and_check) 2026-04-30 16:30:06 [ 434 ] DEBUG : Stderr:Pulling n2 ... status: image is up to date for a... (cluster.py:123, run_and_check) 2026-04-30 16:30:06 [ 434 ] DEBUG : Stderr:Pulling n3 ... done (cluster.py:123, run_and_check) 2026-04-30 16:30:06 [ 434 ] DEBUG : Stderr:Pulling n2 ... done (cluster.py:123, run_and_check) 2026-04-30 16:30:06 [ 434 ] DEBUG : ('Trying to create ClickHouse instance by command %s', 'docker-compose --env-file /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0_gw4/.env --project-name roottestdistributedloadbalancing_gw4 --file /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0_gw4/n1/docker-compose.yml --file /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0_gw4/n2/docker-compose.yml --file /ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0_gw4/n3/docker-compose.yml up -d --no-recreate') (cluster.py:3146, start) 2026-04-30 16:30:06 [ 434 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0_gw4/.env', '--project-name', 'roottestdistributedloadbalancing_gw4', '--file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0_gw4/n1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0_gw4/n2/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0_gw4/n3/docker-compose.yml', 'up', '-d', '--no-recreate'] (cluster.py:113, run_and_check) 2026-04-30 16:30:13 [ 434 ] DEBUG : Stderr:Creating network "roottestdistributedloadbalancing_gw4_default" with the default driver (cluster.py:123, run_and_check) 2026-04-30 16:30:13 [ 434 ] DEBUG : Stderr:Creating roottestdistributedloadbalancing_gw4_n3_1 ... (cluster.py:123, run_and_check) 2026-04-30 16:30:13 [ 434 ] DEBUG : Stderr:Creating roottestdistributedloadbalancing_gw4_n2_1 ... (cluster.py:123, run_and_check) 2026-04-30 16:30:13 [ 434 ] DEBUG : Stderr:Creating roottestdistributedloadbalancing_gw4_n1_1 ... (cluster.py:123, run_and_check) 2026-04-30 16:30:13 [ 434 ] DEBUG : Stderr:Creating roottestdistributedloadbalancing_gw4_n1_1 ... done (cluster.py:123, run_and_check) 2026-04-30 16:30:13 [ 434 ] DEBUG : Stderr:Creating roottestdistributedloadbalancing_gw4_n2_1 ... done (cluster.py:123, run_and_check) 2026-04-30 16:30:13 [ 434 ] DEBUG : Stderr:Creating roottestdistributedloadbalancing_gw4_n3_1 ... done (cluster.py:123, run_and_check) 2026-04-30 16:30:13 [ 434 ] DEBUG : ClickHouse instance created (cluster.py:3154, start) 2026-04-30 16:30:13 [ 434 ] DEBUG : get_instance_ip instance_name=n1 (cluster.py:2135, get_instance_ip) 2026-04-30 16:30:13 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestdistributedloadbalancing_gw4_n1_1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:13 [ 434 ] DEBUG : Waiting for ClickHouse start in n1, ip: 172.16.8.2... (cluster.py:3161, start) 2026-04-30 16:30:13 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestdistributedloadbalancing_gw4_n1_1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:13 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/81a3c67642e3e0b93cd3c1c3a7bcd7c63ffcb997ee639b3836c0885f25efcf63/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:13 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/81a3c67642e3e0b93cd3c1c3a7bcd7c63ffcb997ee639b3836c0885f25efcf63/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:13 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/81a3c67642e3e0b93cd3c1c3a7bcd7c63ffcb997ee639b3836c0885f25efcf63/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:14 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/81a3c67642e3e0b93cd3c1c3a7bcd7c63ffcb997ee639b3836c0885f25efcf63/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:14 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/81a3c67642e3e0b93cd3c1c3a7bcd7c63ffcb997ee639b3836c0885f25efcf63/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:14 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/81a3c67642e3e0b93cd3c1c3a7bcd7c63ffcb997ee639b3836c0885f25efcf63/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:14 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/81a3c67642e3e0b93cd3c1c3a7bcd7c63ffcb997ee639b3836c0885f25efcf63/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:14 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/81a3c67642e3e0b93cd3c1c3a7bcd7c63ffcb997ee639b3836c0885f25efcf63/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:14 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/81a3c67642e3e0b93cd3c1c3a7bcd7c63ffcb997ee639b3836c0885f25efcf63/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:14 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/81a3c67642e3e0b93cd3c1c3a7bcd7c63ffcb997ee639b3836c0885f25efcf63/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:14 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/81a3c67642e3e0b93cd3c1c3a7bcd7c63ffcb997ee639b3836c0885f25efcf63/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:15 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/81a3c67642e3e0b93cd3c1c3a7bcd7c63ffcb997ee639b3836c0885f25efcf63/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:15 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/81a3c67642e3e0b93cd3c1c3a7bcd7c63ffcb997ee639b3836c0885f25efcf63/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:15 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/81a3c67642e3e0b93cd3c1c3a7bcd7c63ffcb997ee639b3836c0885f25efcf63/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:15 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/81a3c67642e3e0b93cd3c1c3a7bcd7c63ffcb997ee639b3836c0885f25efcf63/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:15 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/81a3c67642e3e0b93cd3c1c3a7bcd7c63ffcb997ee639b3836c0885f25efcf63/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:15 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/81a3c67642e3e0b93cd3c1c3a7bcd7c63ffcb997ee639b3836c0885f25efcf63/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:15 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/81a3c67642e3e0b93cd3c1c3a7bcd7c63ffcb997ee639b3836c0885f25efcf63/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:15 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/81a3c67642e3e0b93cd3c1c3a7bcd7c63ffcb997ee639b3836c0885f25efcf63/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:15 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/81a3c67642e3e0b93cd3c1c3a7bcd7c63ffcb997ee639b3836c0885f25efcf63/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:16 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/81a3c67642e3e0b93cd3c1c3a7bcd7c63ffcb997ee639b3836c0885f25efcf63/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:16 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/81a3c67642e3e0b93cd3c1c3a7bcd7c63ffcb997ee639b3836c0885f25efcf63/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:16 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/81a3c67642e3e0b93cd3c1c3a7bcd7c63ffcb997ee639b3836c0885f25efcf63/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:16 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/81a3c67642e3e0b93cd3c1c3a7bcd7c63ffcb997ee639b3836c0885f25efcf63/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:16 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/81a3c67642e3e0b93cd3c1c3a7bcd7c63ffcb997ee639b3836c0885f25efcf63/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:16 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/81a3c67642e3e0b93cd3c1c3a7bcd7c63ffcb997ee639b3836c0885f25efcf63/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:16 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/81a3c67642e3e0b93cd3c1c3a7bcd7c63ffcb997ee639b3836c0885f25efcf63/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:16 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/81a3c67642e3e0b93cd3c1c3a7bcd7c63ffcb997ee639b3836c0885f25efcf63/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:16 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/81a3c67642e3e0b93cd3c1c3a7bcd7c63ffcb997ee639b3836c0885f25efcf63/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:17 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/81a3c67642e3e0b93cd3c1c3a7bcd7c63ffcb997ee639b3836c0885f25efcf63/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:17 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/81a3c67642e3e0b93cd3c1c3a7bcd7c63ffcb997ee639b3836c0885f25efcf63/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:17 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/81a3c67642e3e0b93cd3c1c3a7bcd7c63ffcb997ee639b3836c0885f25efcf63/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:17 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/81a3c67642e3e0b93cd3c1c3a7bcd7c63ffcb997ee639b3836c0885f25efcf63/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:17 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/81a3c67642e3e0b93cd3c1c3a7bcd7c63ffcb997ee639b3836c0885f25efcf63/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:17 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/81a3c67642e3e0b93cd3c1c3a7bcd7c63ffcb997ee639b3836c0885f25efcf63/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:17 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/81a3c67642e3e0b93cd3c1c3a7bcd7c63ffcb997ee639b3836c0885f25efcf63/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:17 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/81a3c67642e3e0b93cd3c1c3a7bcd7c63ffcb997ee639b3836c0885f25efcf63/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:17 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/81a3c67642e3e0b93cd3c1c3a7bcd7c63ffcb997ee639b3836c0885f25efcf63/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:18 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/81a3c67642e3e0b93cd3c1c3a7bcd7c63ffcb997ee639b3836c0885f25efcf63/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:18 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/81a3c67642e3e0b93cd3c1c3a7bcd7c63ffcb997ee639b3836c0885f25efcf63/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:18 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/81a3c67642e3e0b93cd3c1c3a7bcd7c63ffcb997ee639b3836c0885f25efcf63/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:18 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/81a3c67642e3e0b93cd3c1c3a7bcd7c63ffcb997ee639b3836c0885f25efcf63/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:18 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/81a3c67642e3e0b93cd3c1c3a7bcd7c63ffcb997ee639b3836c0885f25efcf63/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:18 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/81a3c67642e3e0b93cd3c1c3a7bcd7c63ffcb997ee639b3836c0885f25efcf63/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:18 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/81a3c67642e3e0b93cd3c1c3a7bcd7c63ffcb997ee639b3836c0885f25efcf63/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:18 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/81a3c67642e3e0b93cd3c1c3a7bcd7c63ffcb997ee639b3836c0885f25efcf63/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:19 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/81a3c67642e3e0b93cd3c1c3a7bcd7c63ffcb997ee639b3836c0885f25efcf63/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:19 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/81a3c67642e3e0b93cd3c1c3a7bcd7c63ffcb997ee639b3836c0885f25efcf63/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:19 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/81a3c67642e3e0b93cd3c1c3a7bcd7c63ffcb997ee639b3836c0885f25efcf63/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:19 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/81a3c67642e3e0b93cd3c1c3a7bcd7c63ffcb997ee639b3836c0885f25efcf63/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:19 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/81a3c67642e3e0b93cd3c1c3a7bcd7c63ffcb997ee639b3836c0885f25efcf63/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:19 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/81a3c67642e3e0b93cd3c1c3a7bcd7c63ffcb997ee639b3836c0885f25efcf63/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:19 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/81a3c67642e3e0b93cd3c1c3a7bcd7c63ffcb997ee639b3836c0885f25efcf63/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:19 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/81a3c67642e3e0b93cd3c1c3a7bcd7c63ffcb997ee639b3836c0885f25efcf63/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:19 [ 434 ] DEBUG : ClickHouse n1 started (cluster.py:3165, start) 2026-04-30 16:30:19 [ 434 ] DEBUG : get_instance_ip instance_name=n2 (cluster.py:2135, get_instance_ip) 2026-04-30 16:30:19 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestdistributedloadbalancing_gw4_n2_1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:19 [ 434 ] DEBUG : Waiting for ClickHouse start in n2, ip: 172.16.8.4... (cluster.py:3161, start) 2026-04-30 16:30:19 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestdistributedloadbalancing_gw4_n2_1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:19 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/66a60e3631a8aff482013c2154a546bcdf123345a56697fccd814d6327961aac/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:19 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/66a60e3631a8aff482013c2154a546bcdf123345a56697fccd814d6327961aac/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:20 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/66a60e3631a8aff482013c2154a546bcdf123345a56697fccd814d6327961aac/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:20 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/66a60e3631a8aff482013c2154a546bcdf123345a56697fccd814d6327961aac/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:20 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/66a60e3631a8aff482013c2154a546bcdf123345a56697fccd814d6327961aac/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:20 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/66a60e3631a8aff482013c2154a546bcdf123345a56697fccd814d6327961aac/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:20 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/66a60e3631a8aff482013c2154a546bcdf123345a56697fccd814d6327961aac/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:20 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/66a60e3631a8aff482013c2154a546bcdf123345a56697fccd814d6327961aac/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:20 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/66a60e3631a8aff482013c2154a546bcdf123345a56697fccd814d6327961aac/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:20 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/66a60e3631a8aff482013c2154a546bcdf123345a56697fccd814d6327961aac/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:21 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/66a60e3631a8aff482013c2154a546bcdf123345a56697fccd814d6327961aac/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:21 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/66a60e3631a8aff482013c2154a546bcdf123345a56697fccd814d6327961aac/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:21 [ 434 ] DEBUG : ClickHouse n2 started (cluster.py:3165, start) 2026-04-30 16:30:21 [ 434 ] DEBUG : get_instance_ip instance_name=n3 (cluster.py:2135, get_instance_ip) 2026-04-30 16:30:21 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestdistributedloadbalancing_gw4_n3_1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:21 [ 434 ] DEBUG : Waiting for ClickHouse start in n3, ip: 172.16.8.3... (cluster.py:3161, start) 2026-04-30 16:30:21 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestdistributedloadbalancing_gw4_n3_1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:21 [ 434 ] DEBUG : http://localhost:None "GET /v1.42/containers/eecacd01fc21c75ba7b662dd14bd7290ba3c62c851926cb5fc9ecd30d187bb15/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:30:21 [ 434 ] DEBUG : ClickHouse n3 started (cluster.py:3165, start) 2026-04-30 16:30:21 [ 434 ] DEBUG : Executing query DROP TABLE IF EXISTS data on n1 (cluster.py:3602, query) 2026-04-30 16:30:21 [ 434 ] DEBUG : Executing query DROP TABLE IF EXISTS dist on n1 (cluster.py:3602, query) 2026-04-30 16:30:22 [ 434 ] DEBUG : Executing query CREATE TABLE data (key Int) Engine=Memory() on n1 (cluster.py:3602, query) 2026-04-30 16:30:23 [ 434 ] DEBUG : Executing query CREATE TABLE dist AS data Engine=Distributed( replicas_cluster, currentDatabase(), data) on n1 (cluster.py:3602, query) 2026-04-30 16:30:23 [ 434 ] DEBUG : Executing query CREATE TABLE dist_priority AS data Engine=Distributed( replicas_priority_cluster, currentDatabase(), data) on n1 (cluster.py:3602, query) 2026-04-30 16:30:24 [ 434 ] DEBUG : Executing query CREATE TABLE dist_priority_negative AS data Engine=Distributed( replicas_priority_negative_cluster, currentDatabase(), data) on n1 (cluster.py:3602, query) 2026-04-30 16:30:25 [ 434 ] DEBUG : Executing query DROP TABLE IF EXISTS data on n2 (cluster.py:3602, query) 2026-04-30 16:30:25 [ 434 ] DEBUG : Executing query DROP TABLE IF EXISTS dist on n2 (cluster.py:3602, query) 2026-04-30 16:30:26 [ 434 ] DEBUG : Executing query CREATE TABLE data (key Int) Engine=Memory() on n2 (cluster.py:3602, query) 2026-04-30 16:30:27 [ 434 ] DEBUG : Executing query CREATE TABLE dist AS data Engine=Distributed( replicas_cluster, currentDatabase(), data) on n2 (cluster.py:3602, query) 2026-04-30 16:30:27 [ 434 ] DEBUG : Executing query CREATE TABLE dist_priority AS data Engine=Distributed( replicas_priority_cluster, currentDatabase(), data) on n2 (cluster.py:3602, query) 2026-04-30 16:30:28 [ 434 ] DEBUG : Executing query CREATE TABLE dist_priority_negative AS data Engine=Distributed( replicas_priority_negative_cluster, currentDatabase(), data) on n2 (cluster.py:3602, query) 2026-04-30 16:30:29 [ 434 ] DEBUG : Executing query DROP TABLE IF EXISTS data on n3 (cluster.py:3602, query) 2026-04-30 16:30:30 [ 434 ] DEBUG : Executing query DROP TABLE IF EXISTS dist on n3 (cluster.py:3602, query) 2026-04-30 16:30:30 [ 434 ] DEBUG : Executing query CREATE TABLE data (key Int) Engine=Memory() on n3 (cluster.py:3602, query) 2026-04-30 16:30:31 [ 434 ] DEBUG : Executing query CREATE TABLE dist AS data Engine=Distributed( replicas_cluster, currentDatabase(), data) on n3 (cluster.py:3602, query) 2026-04-30 16:30:32 [ 434 ] DEBUG : Executing query CREATE TABLE dist_priority AS data Engine=Distributed( replicas_priority_cluster, currentDatabase(), data) on n3 (cluster.py:3602, query) 2026-04-30 16:30:33 [ 434 ] DEBUG : Executing query CREATE TABLE dist_priority_negative AS data Engine=Distributed( replicas_priority_negative_cluster, currentDatabase(), data) on n3 (cluster.py:3602, query) 2026-04-30 16:30:33 [ 434 ] DEBUG : Executing query SYSTEM RELOAD CONFIG on n1 (cluster.py:3602, query) 2026-04-30 16:30:34 [ 434 ] DEBUG : Executing query SYSTEM RELOAD CONFIG on n2 (cluster.py:3602, query) 2026-04-30 16:30:35 [ 434 ] DEBUG : Executing query SYSTEM RELOAD CONFIG on n3 (cluster.py:3602, query) ------------------------------ Captured log call ------------------------------- 2026-04-30 16:30:36 [ 434 ] DEBUG : Executing query SELECT * FROM dist on n2 (cluster.py:3602, query) 2026-04-30 16:30:37 [ 434 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0_gw4/.env', '--project-name', 'roottestdistributedloadbalancing_gw4', '--file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0_gw4/n1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0_gw4/n2/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0_gw4/n3/docker-compose.yml', 'pause', 'n1'] (cluster.py:113, run_and_check) 2026-04-30 16:30:39 [ 434 ] DEBUG : Stderr:Pausing roottestdistributedloadbalancing_gw4_n1_1 ... (cluster.py:123, run_and_check) 2026-04-30 16:30:39 [ 434 ] DEBUG : Stderr:Pausing roottestdistributedloadbalancing_gw4_n1_1 ... done (cluster.py:123, run_and_check) 2026-04-30 16:30:39 [ 434 ] DEBUG : Executing query SELECT * FROM dist on n2 (cluster.py:3602, query) 2026-04-30 16:30:42 [ 434 ] DEBUG : Executing query SELECT * FROM dist on n2 (cluster.py:3602, query) 2026-04-30 16:30:43 [ 434 ] DEBUG : Executing query SELECT errors_count FROM system.clusters WHERE cluster = 'replicas_cluster' AND host_name = 'n1' on n2 (cluster.py:3602, query) 2026-04-30 16:30:44 [ 434 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0_gw4/.env', '--project-name', 'roottestdistributedloadbalancing_gw4', '--file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0_gw4/n1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0_gw4/n2/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_distributed_load_balancing/_instances_0_gw4/n3/docker-compose.yml', 'unpause', 'n1'] (cluster.py:113, run_and_check) 2026-04-30 16:30:45 [ 434 ] DEBUG : Stderr:Unpausing roottestdistributedloadbalancing_gw4_n1_1 ... (cluster.py:123, run_and_check) 2026-04-30 16:30:45 [ 434 ] DEBUG : Stderr:Unpausing roottestdistributedloadbalancing_gw4_n1_1 ... done (cluster.py:123, run_and_check) 2026-04-30 16:30:45 [ 434 ] DEBUG : Executing query SELECT * FROM dist on n2 (cluster.py:3602, query) 2026-04-30 16:30:46 [ 434 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n1 (cluster.py:3602, query) 2026-04-30 16:30:50 [ 434 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n2 (cluster.py:3602, query) 2026-04-30 16:30:52 [ 434 ] DEBUG : Executing query SYSTEM FLUSH LOGS on n3 (cluster.py:3602, query) 2026-04-30 16:30:54 [ 434 ] DEBUG : Executing query SELECT hostName() FROM cluster(shards_cluster, system.query_log) WHERE initial_query_id = '5fd873420511441f9c335be92af970bd' AND is_initial_query = 0 AND type = 'QueryFinish' ORDER BY event_date DESC, event_time DESC LIMIT 1 on n2 (cluster.py:3602, query) ____________________________ test_receive_timeout1 _____________________________ [gw0] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_receive_timeout1(started_cluster): # Check the situation when first two replicas get receive timeout # in establishing connection, but the third replica is ok. update_configs( node_1_sleep_in_send_tables_status=3000, node_2_sleep_in_send_tables_status=3000, node_3_sleep_in_send_data=1000, ) > check_query(expected_replica="node_3", receive_timeout=2) test_hedged_requests/test.py:327: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test_hedged_requests/test.py:72: in check_query NODES["node"].restart_clickhouse() helpers/cluster.py:4055: in restart_clickhouse self.start_clickhouse(stop_start_wait_sec) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = start_wait_sec = 60, retry_start = True, expected_to_fail = False def start_clickhouse( self, start_wait_sec=60, retry_start=True, expected_to_fail=False ): if not self.stay_alive: raise Exception( "ClickHouse can be started again only with stay_alive=True instance" ) start_time = time.time() time_to_sleep = 0.5 while start_time + start_wait_sec >= time.time(): # sometimes after SIGKILL (hard reset) server may refuse to start for some time # for different reasons. pid = self.get_process_pid("clickhouse") if pid is None: logging.debug("No clickhouse process running. Start new one.") self.exec_in_container( ["bash", "-c", "{} --daemon".format(self.clickhouse_start_command)], user=str(os.getuid()), ) if expected_to_fail: self.wait_start_failed(start_wait_sec + start_time - time.time()) return time.sleep(1) continue else: logging.debug("Clickhouse process running.") if expected_to_fail: raise Exception("ClickHouse was expected not to be running.") try: self.wait_start(start_wait_sec + start_time - time.time()) return except Exception as e: logging.warning( f"Current start attempt failed. Will kill {pid} just in case." ) self.exec_in_container( ["bash", "-c", f"kill -9 {pid}"], user="root", nothrow=True ) if not retry_start: raise time.sleep(time_to_sleep) > raise Exception("Cannot start ClickHouse, see additional info in logs") E Exception: Cannot start ClickHouse, see additional info in logs helpers/cluster.py:3992: Exception ------------------------------ Captured log call ------------------------------- 2026-04-30 16:38:10 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1_1 detach:False nothrow:False cmd: ['bash', '-c', "echo '\n \n \n 3000\n 0\n 0\n \n \n' > /etc/clickhouse-server/users.d/users1.xml"] (cluster.py:2173, exec_in_container) 2026-04-30 16:38:10 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1_1', 'bash', '-c', "echo '\n \n \n 3000\n 0\n 0\n \n \n' > /etc/clickhouse-server/users.d/users1.xml"] (cluster.py:113, run_and_check) 2026-04-30 16:38:10 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_2_1 detach:False nothrow:False cmd: ['bash', '-c', "echo '\n \n \n 3000\n 0\n 0\n \n \n' > /etc/clickhouse-server/users.d/users1.xml"] (cluster.py:2173, exec_in_container) 2026-04-30 16:38:10 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_2_1', 'bash', '-c', "echo '\n \n \n 3000\n 0\n 0\n \n \n' > /etc/clickhouse-server/users.d/users1.xml"] (cluster.py:113, run_and_check) 2026-04-30 16:38:11 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_3_1 detach:False nothrow:False cmd: ['bash', '-c', "echo '\n \n \n 0\n 1000\n 0\n \n \n' > /etc/clickhouse-server/users.d/users1.xml"] (cluster.py:2173, exec_in_container) 2026-04-30 16:38:11 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_3_1', 'bash', '-c', "echo '\n \n \n 0\n 1000\n 0\n \n \n' > /etc/clickhouse-server/users.d/users1.xml"] (cluster.py:113, run_and_check) 2026-04-30 16:38:12 [ 422 ] DEBUG : Executing query SELECT value FROM system.settings WHERE name='sleep_in_send_tables_status_ms' on node_1 via HTTP interface (cluster.py:3850, http_query_and_get_answer_with_error) 2026-04-30 16:38:12 [ 422 ] DEBUG : Starting new HTTP connection (1): 172.16.6.4:8123 (connectionpool.py:245, _new_conn) 2026-04-30 16:38:12 [ 422 ] DEBUG : http://172.16.6.4:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_in_send_tables_status_ms%27 HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:38:12 [ 422 ] DEBUG : Executing query SELECT value FROM system.settings WHERE name='sleep_in_send_data_ms' on node_1 via HTTP interface (cluster.py:3850, http_query_and_get_answer_with_error) 2026-04-30 16:38:12 [ 422 ] DEBUG : Starting new HTTP connection (1): 172.16.6.4:8123 (connectionpool.py:245, _new_conn) 2026-04-30 16:38:13 [ 422 ] DEBUG : http://172.16.6.4:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_in_send_data_ms%27 HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:38:13 [ 422 ] DEBUG : Executing query SELECT value FROM system.settings WHERE name='sleep_after_receiving_query_ms' on node_1 via HTTP interface (cluster.py:3850, http_query_and_get_answer_with_error) 2026-04-30 16:38:13 [ 422 ] DEBUG : Starting new HTTP connection (1): 172.16.6.4:8123 (connectionpool.py:245, _new_conn) 2026-04-30 16:38:13 [ 422 ] DEBUG : http://172.16.6.4:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_after_receiving_query_ms%27 HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:38:14 [ 422 ] DEBUG : Executing query SELECT value FROM system.settings WHERE name='sleep_in_send_tables_status_ms' on node_2 via HTTP interface (cluster.py:3850, http_query_and_get_answer_with_error) 2026-04-30 16:38:14 [ 422 ] DEBUG : Starting new HTTP connection (1): 172.16.6.3:8123 (connectionpool.py:245, _new_conn) 2026-04-30 16:38:14 [ 422 ] DEBUG : http://172.16.6.3:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_in_send_tables_status_ms%27 HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:38:14 [ 422 ] DEBUG : Executing query SELECT value FROM system.settings WHERE name='sleep_in_send_data_ms' on node_2 via HTTP interface (cluster.py:3850, http_query_and_get_answer_with_error) 2026-04-30 16:38:14 [ 422 ] DEBUG : Starting new HTTP connection (1): 172.16.6.3:8123 (connectionpool.py:245, _new_conn) 2026-04-30 16:38:14 [ 422 ] DEBUG : http://172.16.6.3:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_in_send_data_ms%27 HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:38:15 [ 422 ] DEBUG : Executing query SELECT value FROM system.settings WHERE name='sleep_after_receiving_query_ms' on node_2 via HTTP interface (cluster.py:3850, http_query_and_get_answer_with_error) 2026-04-30 16:38:15 [ 422 ] DEBUG : Starting new HTTP connection (1): 172.16.6.3:8123 (connectionpool.py:245, _new_conn) 2026-04-30 16:38:15 [ 422 ] DEBUG : http://172.16.6.3:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_after_receiving_query_ms%27 HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:38:15 [ 422 ] DEBUG : Executing query SELECT value FROM system.settings WHERE name='sleep_in_send_tables_status_ms' on node_3 via HTTP interface (cluster.py:3850, http_query_and_get_answer_with_error) 2026-04-30 16:38:15 [ 422 ] DEBUG : Starting new HTTP connection (1): 172.16.6.5:8123 (connectionpool.py:245, _new_conn) 2026-04-30 16:38:15 [ 422 ] DEBUG : http://172.16.6.5:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_in_send_tables_status_ms%27 HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:38:15 [ 422 ] DEBUG : Executing query SELECT value FROM system.settings WHERE name='sleep_in_send_data_ms' on node_3 via HTTP interface (cluster.py:3850, http_query_and_get_answer_with_error) 2026-04-30 16:38:15 [ 422 ] DEBUG : Starting new HTTP connection (1): 172.16.6.5:8123 (connectionpool.py:245, _new_conn) 2026-04-30 16:38:15 [ 422 ] DEBUG : http://172.16.6.5:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_in_send_data_ms%27 HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:38:15 [ 422 ] DEBUG : Executing query SELECT value FROM system.settings WHERE name='sleep_after_receiving_query_ms' on node_3 via HTTP interface (cluster.py:3850, http_query_and_get_answer_with_error) 2026-04-30 16:38:15 [ 422 ] DEBUG : Starting new HTTP connection (1): 172.16.6.5:8123 (connectionpool.py:245, _new_conn) 2026-04-30 16:38:16 [ 422 ] DEBUG : http://172.16.6.5:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_after_receiving_query_ms%27 HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:38:16 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:True cmd: ['bash', '-c', 'ps -C clickhouse'] (cluster.py:2173, exec_in_container) 2026-04-30 16:38:16 [ 422 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', 'ps -C clickhouse'] (cluster.py:113, run_and_check) 2026-04-30 16:38:17 [ 422 ] DEBUG : Stdout: PID TTY TIME CMD (cluster.py:121, run_and_check) 2026-04-30 16:38:17 [ 422 ] DEBUG : Stdout: 7400 ? 00:04:33 clickhouse (cluster.py:121, run_and_check) 2026-04-30 16:38:17 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', 'pkill clickhouse'] (cluster.py:2173, exec_in_container) 2026-04-30 16:38:17 [ 422 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', 'pkill clickhouse'] (cluster.py:113, run_and_check) 2026-04-30 16:38:17 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:38:17 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:38:18 [ 422 ] DEBUG : Stdout:7400 (cluster.py:121, run_and_check) 2026-04-30 16:38:19 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:38:19 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:38:20 [ 422 ] DEBUG : Stdout:7400 (cluster.py:121, run_and_check) 2026-04-30 16:38:21 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:38:21 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:38:22 [ 422 ] DEBUG : Stdout:7400 (cluster.py:121, run_and_check) 2026-04-30 16:38:23 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:38:23 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:38:24 [ 422 ] DEBUG : Stdout:7400 (cluster.py:121, run_and_check) 2026-04-30 16:38:25 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:38:25 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:38:25 [ 422 ] DEBUG : Stdout:7400 (cluster.py:121, run_and_check) 2026-04-30 16:38:26 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:38:26 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:38:27 [ 422 ] DEBUG : Stdout:7400 (cluster.py:121, run_and_check) 2026-04-30 16:38:28 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:38:28 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:38:29 [ 422 ] DEBUG : Stdout:7400 (cluster.py:121, run_and_check) 2026-04-30 16:38:30 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:38:30 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:38:31 [ 422 ] DEBUG : Stdout:7400 (cluster.py:121, run_and_check) 2026-04-30 16:38:32 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:38:32 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:38:33 [ 422 ] DEBUG : Stdout:7400 (cluster.py:121, run_and_check) 2026-04-30 16:38:34 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:38:34 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:38:35 [ 422 ] DEBUG : Stdout:7400 (cluster.py:121, run_and_check) 2026-04-30 16:38:36 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:38:36 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:38:36 [ 422 ] DEBUG : Stdout:7400 (cluster.py:121, run_and_check) 2026-04-30 16:38:37 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:38:37 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:38:38 [ 422 ] DEBUG : Stdout:7400 (cluster.py:121, run_and_check) 2026-04-30 16:38:39 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:38:39 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:38:40 [ 422 ] DEBUG : Stdout:7400 (cluster.py:121, run_and_check) 2026-04-30 16:38:40 [ 422 ] DEBUG : Stdout:8335 (cluster.py:121, run_and_check) 2026-04-30 16:38:41 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:38:41 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:38:42 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:38:42 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:38:43 [ 422 ] DEBUG : No clickhouse process running. Start new one. (cluster.py:3964, start_clickhouse) 2026-04-30 16:38:43 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', 'clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon'] (cluster.py:2173, exec_in_container) 2026-04-30 16:38:43 [ 422 ] DEBUG : Command:['docker', 'exec', '-u', '0', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', 'clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon'] (cluster.py:113, run_and_check) 2026-04-30 16:38:46 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:38:46 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:38:47 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:38:47 [ 422 ] DEBUG : Clickhouse process running. (cluster.py:3975, start_clickhouse) 2026-04-30 16:38:47 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:38:47 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:38:48 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:38:48 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:38:49 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:38:50 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:38:52 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:38:53 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:38:55 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:38:56 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:38:58 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:38:59 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:39:01 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:39:03 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:39:03 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:39:04 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:39:04 [ 422 ] WARNING : ERROR Client failed! Return code: 210, stderr: Code: 210. DB::NetException: Connection refused (172.16.6.2:9000). (NETWORK_ERROR) (cluster.py:4008, wait_start) 2026-04-30 16:39:04 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:39:04 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:39:04 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:39:04 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:39:06 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:39:08 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:39:09 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:39:11 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:39:13 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:39:14 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:39:15 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:39:17 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:39:19 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:39:21 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:39:21 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:39:21 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:39:21 [ 422 ] WARNING : ERROR Client failed! Return code: 210, stderr: Code: 210. DB::NetException: Connection refused (172.16.6.2:9000). (NETWORK_ERROR) (cluster.py:4008, wait_start) 2026-04-30 16:39:21 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:39:21 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:39:22 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:39:22 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:39:24 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:39:25 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:39:27 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:39:28 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:39:29 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:39:30 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:39:32 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:39:33 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:39:35 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:39:36 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:39:36 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:39:37 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:39:37 [ 422 ] WARNING : ERROR Client failed! Return code: 210, stderr: Code: 210. DB::NetException: Connection refused (172.16.6.2:9000). (NETWORK_ERROR) (cluster.py:4008, wait_start) 2026-04-30 16:39:37 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:39:37 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:39:38 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:39:38 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:39:39 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:39:41 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:39:42 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:39:44 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:39:45 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:39:47 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:39:48 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:39:50 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:39:51 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:39:53 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:39:53 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:39:54 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:39:54 [ 422 ] WARNING : ERROR Client failed! Return code: 210, stderr: Code: 210. DB::NetException: Connection refused (172.16.6.2:9000). (NETWORK_ERROR) (cluster.py:4008, wait_start) 2026-04-30 16:39:54 [ 422 ] ERROR : No time left to start. But process is still running. Will dump threads. (cluster.py:4013, wait_start) 2026-04-30 16:39:54 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:True cmd: ['bash', '-c', 'ps -C clickhouse'] (cluster.py:2173, exec_in_container) 2026-04-30 16:39:54 [ 422 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', 'ps -C clickhouse'] (cluster.py:113, run_and_check) 2026-04-30 16:39:55 [ 422 ] DEBUG : Stdout: PID TTY TIME CMD (cluster.py:121, run_and_check) 2026-04-30 16:39:55 [ 422 ] DEBUG : Stdout: 8378 ? 00:04:23 clickhouse (cluster.py:121, run_and_check) 2026-04-30 16:39:55 [ 422 ] INFO : PS RESULT: PID TTY TIME CMD 8378 ? 00:04:23 clickhouse (cluster.py:4019, wait_start) 2026-04-30 16:39:55 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:39:55 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:39:56 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:39:56 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "gdb -batch -ex 'thread apply all bt full' -p 8378"] (cluster.py:2173, exec_in_container) 2026-04-30 16:39:56 [ 422 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "gdb -batch -ex 'thread apply all bt full' -p 8378"] (cluster.py:113, run_and_check) 2026-04-30 16:44:56 [ 422 ] WARNING : Current start attempt failed. Will kill 8378 just in case. (cluster.py:3982, start_clickhouse) 2026-04-30 16:44:56 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:True cmd: ['bash', '-c', 'kill -9 8378'] (cluster.py:2173, exec_in_container) 2026-04-30 16:44:56 [ 422 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', 'kill -9 8378'] (cluster.py:113, run_and_check) ____________________________ test_receive_timeout2 _____________________________ [gw0] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_receive_timeout2(started_cluster): # Check the situation when first replica get receive timeout # in packet receiving but there are replicas in process of # connection establishing. update_configs( node_1_sleep_in_send_data=5000, node_2_sleep_in_send_tables_status=2000, node_3_sleep_in_send_tables_status=2000, ) > check_query(expected_replica="node_2", receive_timeout=3) test_hedged_requests/test.py:340: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test_hedged_requests/test.py:72: in check_query NODES["node"].restart_clickhouse() helpers/cluster.py:4055: in restart_clickhouse self.start_clickhouse(stop_start_wait_sec) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = start_wait_sec = 60, retry_start = True, expected_to_fail = False def start_clickhouse( self, start_wait_sec=60, retry_start=True, expected_to_fail=False ): if not self.stay_alive: raise Exception( "ClickHouse can be started again only with stay_alive=True instance" ) start_time = time.time() time_to_sleep = 0.5 while start_time + start_wait_sec >= time.time(): # sometimes after SIGKILL (hard reset) server may refuse to start for some time # for different reasons. pid = self.get_process_pid("clickhouse") if pid is None: logging.debug("No clickhouse process running. Start new one.") self.exec_in_container( ["bash", "-c", "{} --daemon".format(self.clickhouse_start_command)], user=str(os.getuid()), ) if expected_to_fail: self.wait_start_failed(start_wait_sec + start_time - time.time()) return time.sleep(1) continue else: logging.debug("Clickhouse process running.") if expected_to_fail: raise Exception("ClickHouse was expected not to be running.") try: self.wait_start(start_wait_sec + start_time - time.time()) return except Exception as e: logging.warning( f"Current start attempt failed. Will kill {pid} just in case." ) self.exec_in_container( ["bash", "-c", f"kill -9 {pid}"], user="root", nothrow=True ) if not retry_start: raise time.sleep(time_to_sleep) > raise Exception("Cannot start ClickHouse, see additional info in logs") E Exception: Cannot start ClickHouse, see additional info in logs helpers/cluster.py:3992: Exception ------------------------------ Captured log call ------------------------------- 2026-04-30 16:44:58 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1_1 detach:False nothrow:False cmd: ['bash', '-c', "echo '\n \n \n 0\n 5000\n 0\n \n \n' > /etc/clickhouse-server/users.d/users1.xml"] (cluster.py:2173, exec_in_container) 2026-04-30 16:44:58 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1_1', 'bash', '-c', "echo '\n \n \n 0\n 5000\n 0\n \n \n' > /etc/clickhouse-server/users.d/users1.xml"] (cluster.py:113, run_and_check) 2026-04-30 16:44:59 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_2_1 detach:False nothrow:False cmd: ['bash', '-c', "echo '\n \n \n 2000\n 0\n 0\n \n \n' > /etc/clickhouse-server/users.d/users1.xml"] (cluster.py:2173, exec_in_container) 2026-04-30 16:44:59 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_2_1', 'bash', '-c', "echo '\n \n \n 2000\n 0\n 0\n \n \n' > /etc/clickhouse-server/users.d/users1.xml"] (cluster.py:113, run_and_check) 2026-04-30 16:44:59 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_3_1 detach:False nothrow:False cmd: ['bash', '-c', "echo '\n \n \n 2000\n 0\n 0\n \n \n' > /etc/clickhouse-server/users.d/users1.xml"] (cluster.py:2173, exec_in_container) 2026-04-30 16:44:59 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_3_1', 'bash', '-c', "echo '\n \n \n 2000\n 0\n 0\n \n \n' > /etc/clickhouse-server/users.d/users1.xml"] (cluster.py:113, run_and_check) 2026-04-30 16:45:00 [ 422 ] DEBUG : Executing query SELECT value FROM system.settings WHERE name='sleep_in_send_tables_status_ms' on node_1 via HTTP interface (cluster.py:3850, http_query_and_get_answer_with_error) 2026-04-30 16:45:00 [ 422 ] DEBUG : Starting new HTTP connection (1): 172.16.6.4:8123 (connectionpool.py:245, _new_conn) 2026-04-30 16:45:00 [ 422 ] DEBUG : http://172.16.6.4:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_in_send_tables_status_ms%27 HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:45:00 [ 422 ] DEBUG : Executing query SELECT value FROM system.settings WHERE name='sleep_in_send_data_ms' on node_1 via HTTP interface (cluster.py:3850, http_query_and_get_answer_with_error) 2026-04-30 16:45:00 [ 422 ] DEBUG : Starting new HTTP connection (1): 172.16.6.4:8123 (connectionpool.py:245, _new_conn) 2026-04-30 16:45:01 [ 422 ] DEBUG : http://172.16.6.4:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_in_send_data_ms%27 HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:45:01 [ 422 ] DEBUG : Executing query SELECT value FROM system.settings WHERE name='sleep_after_receiving_query_ms' on node_1 via HTTP interface (cluster.py:3850, http_query_and_get_answer_with_error) 2026-04-30 16:45:01 [ 422 ] DEBUG : Starting new HTTP connection (1): 172.16.6.4:8123 (connectionpool.py:245, _new_conn) 2026-04-30 16:45:02 [ 422 ] DEBUG : http://172.16.6.4:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_after_receiving_query_ms%27 HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:45:02 [ 422 ] DEBUG : Executing query SELECT value FROM system.settings WHERE name='sleep_in_send_tables_status_ms' on node_2 via HTTP interface (cluster.py:3850, http_query_and_get_answer_with_error) 2026-04-30 16:45:02 [ 422 ] DEBUG : Starting new HTTP connection (1): 172.16.6.3:8123 (connectionpool.py:245, _new_conn) 2026-04-30 16:45:02 [ 422 ] DEBUG : http://172.16.6.3:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_in_send_tables_status_ms%27 HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:45:02 [ 422 ] DEBUG : Executing query SELECT value FROM system.settings WHERE name='sleep_in_send_data_ms' on node_2 via HTTP interface (cluster.py:3850, http_query_and_get_answer_with_error) 2026-04-30 16:45:02 [ 422 ] DEBUG : Starting new HTTP connection (1): 172.16.6.3:8123 (connectionpool.py:245, _new_conn) 2026-04-30 16:45:03 [ 422 ] DEBUG : http://172.16.6.3:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_in_send_data_ms%27 HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:45:03 [ 422 ] DEBUG : Executing query SELECT value FROM system.settings WHERE name='sleep_after_receiving_query_ms' on node_2 via HTTP interface (cluster.py:3850, http_query_and_get_answer_with_error) 2026-04-30 16:45:03 [ 422 ] DEBUG : Starting new HTTP connection (1): 172.16.6.3:8123 (connectionpool.py:245, _new_conn) 2026-04-30 16:45:03 [ 422 ] DEBUG : http://172.16.6.3:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_after_receiving_query_ms%27 HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:45:03 [ 422 ] DEBUG : Executing query SELECT value FROM system.settings WHERE name='sleep_in_send_tables_status_ms' on node_3 via HTTP interface (cluster.py:3850, http_query_and_get_answer_with_error) 2026-04-30 16:45:03 [ 422 ] DEBUG : Starting new HTTP connection (1): 172.16.6.5:8123 (connectionpool.py:245, _new_conn) 2026-04-30 16:45:03 [ 422 ] DEBUG : http://172.16.6.5:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_in_send_tables_status_ms%27 HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:45:03 [ 422 ] DEBUG : Executing query SELECT value FROM system.settings WHERE name='sleep_in_send_data_ms' on node_3 via HTTP interface (cluster.py:3850, http_query_and_get_answer_with_error) 2026-04-30 16:45:03 [ 422 ] DEBUG : Starting new HTTP connection (1): 172.16.6.5:8123 (connectionpool.py:245, _new_conn) 2026-04-30 16:45:03 [ 422 ] DEBUG : http://172.16.6.5:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_in_send_data_ms%27 HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:45:03 [ 422 ] DEBUG : Executing query SELECT value FROM system.settings WHERE name='sleep_after_receiving_query_ms' on node_3 via HTTP interface (cluster.py:3850, http_query_and_get_answer_with_error) 2026-04-30 16:45:03 [ 422 ] DEBUG : Starting new HTTP connection (1): 172.16.6.5:8123 (connectionpool.py:245, _new_conn) 2026-04-30 16:45:04 [ 422 ] DEBUG : http://172.16.6.5:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_after_receiving_query_ms%27 HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:45:04 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:True cmd: ['bash', '-c', 'ps -C clickhouse'] (cluster.py:2173, exec_in_container) 2026-04-30 16:45:04 [ 422 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', 'ps -C clickhouse'] (cluster.py:113, run_and_check) 2026-04-30 16:45:04 [ 422 ] DEBUG : Stdout: PID TTY TIME CMD (cluster.py:121, run_and_check) 2026-04-30 16:45:04 [ 422 ] DEBUG : Stdout: 8378 ? 00:04:34 clickhouse (cluster.py:121, run_and_check) 2026-04-30 16:45:04 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', 'pkill clickhouse'] (cluster.py:2173, exec_in_container) 2026-04-30 16:45:04 [ 422 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', 'pkill clickhouse'] (cluster.py:113, run_and_check) 2026-04-30 16:45:05 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:45:05 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:45:06 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:45:07 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:45:07 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:45:07 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:45:08 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:45:08 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:45:09 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:45:10 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:45:10 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:45:11 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:45:12 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:45:12 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:45:12 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:45:13 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:45:13 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:45:14 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:45:15 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:45:15 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:45:15 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:45:16 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:45:16 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:45:17 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:45:18 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:45:18 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:45:19 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:45:20 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:45:20 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:45:20 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:45:21 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:45:21 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:45:21 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:45:22 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:45:22 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:45:23 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:45:24 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:45:24 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:45:25 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:45:26 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:45:26 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:45:26 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:45:27 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:45:27 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:45:28 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:45:29 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:45:29 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:45:29 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:45:30 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:45:30 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:45:31 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:45:32 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:45:32 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:45:32 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:45:33 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:45:33 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:45:34 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:45:35 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:45:35 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:45:35 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:45:36 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:45:36 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:45:37 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:45:38 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:45:38 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:45:38 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:45:39 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:45:39 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:45:40 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:45:41 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:45:41 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:45:41 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:45:42 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:45:42 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:45:43 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:45:44 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:45:44 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:45:45 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:45:46 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:45:46 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:45:46 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:45:47 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:45:47 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:45:48 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:45:49 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:45:49 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:45:49 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:45:50 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:45:50 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:45:51 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:45:52 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:45:52 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:45:52 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:45:53 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:45:53 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:45:54 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:45:55 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:45:55 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:45:55 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:45:56 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:45:56 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:45:56 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:45:57 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:45:57 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:45:58 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:45:59 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:45:59 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:45:59 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:46:00 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:46:00 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:46:01 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:46:02 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:46:02 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:46:02 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:46:03 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:46:03 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:46:04 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:46:05 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:46:05 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:46:05 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:46:06 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:46:06 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:46:07 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:46:07 [ 422 ] WARNING : Force kill clickhouse in stop_clickhouse. ps:8378 (cluster.py:3926, stop_clickhouse) 2026-04-30 16:46:07 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "gdb -batch -ex 'thread apply all bt full' -p 8378 > /ClickHouse/tests/integration/test_hedged_requests/_instances_0_gw0/node/logs/stdout.log"] (cluster.py:2173, exec_in_container) 2026-04-30 16:46:07 [ 422 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "gdb -batch -ex 'thread apply all bt full' -p 8378 > /ClickHouse/tests/integration/test_hedged_requests/_instances_0_gw0/node/logs/stdout.log"] (cluster.py:113, run_and_check) 2026-04-30 16:46:07 [ 422 ] DEBUG : Stderr:bash: line 1: /ClickHouse/tests/integration/test_hedged_requests/_instances_0_gw0/node/logs/stdout.log: No such file or directory (cluster.py:123, run_and_check) 2026-04-30 16:46:07 [ 422 ] DEBUG : Exitcode:1 (cluster.py:125, run_and_check) 2026-04-30 16:46:07 [ 422 ] WARNING : Stop ClickHouse raised an error Command ['docker', 'exec', '-u', 'root', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "gdb -batch -ex 'thread apply all bt full' -p 8378 > /ClickHouse/tests/integration/test_hedged_requests/_instances_0_gw0/node/logs/stdout.log"] return non-zero code 1: bash: line 1: /ClickHouse/tests/integration/test_hedged_requests/_instances_0_gw0/node/logs/stdout.log: No such file or directory (cluster.py:3947, stop_clickhouse) 2026-04-30 16:46:07 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:46:07 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:46:08 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:46:08 [ 422 ] DEBUG : Clickhouse process running. (cluster.py:3975, start_clickhouse) 2026-04-30 16:46:08 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:46:08 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:46:08 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:46:08 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:46:09 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:46:11 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:46:12 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:46:14 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:46:15 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:46:16 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:46:17 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:46:18 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:46:20 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:46:21 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:46:21 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:46:21 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:46:21 [ 422 ] WARNING : ERROR Client failed! Return code: 210, stderr: Code: 210. DB::NetException: Connection refused (172.16.6.2:9000). (NETWORK_ERROR) (cluster.py:4008, wait_start) 2026-04-30 16:46:21 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:46:21 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:46:22 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:46:22 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:46:23 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:46:24 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:46:25 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:46:26 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:46:27 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:46:28 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:46:29 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:46:30 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:46:31 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:46:32 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:46:32 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:46:32 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:46:32 [ 422 ] WARNING : ERROR Client failed! Return code: 210, stderr: Code: 210. DB::NetException: Connection refused (172.16.6.2:9000). (NETWORK_ERROR) (cluster.py:4008, wait_start) 2026-04-30 16:46:32 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:46:32 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:46:33 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:46:33 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:46:34 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:46:35 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:46:36 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:46:37 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:46:38 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:46:39 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:46:40 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:46:41 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:46:42 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:46:43 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:46:43 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:46:44 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:46:44 [ 422 ] WARNING : ERROR Client failed! Return code: 210, stderr: Code: 210. DB::NetException: Connection refused (172.16.6.2:9000). (NETWORK_ERROR) (cluster.py:4008, wait_start) 2026-04-30 16:46:44 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:46:44 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:46:44 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:46:44 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:46:45 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:46:46 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:46:48 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:46:49 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:46:50 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:46:51 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:46:53 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:46:54 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:46:55 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:46:56 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:46:56 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:46:57 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:46:57 [ 422 ] WARNING : ERROR Client failed! Return code: 210, stderr: Code: 210. DB::NetException: Connection refused (172.16.6.2:9000). (NETWORK_ERROR) (cluster.py:4008, wait_start) 2026-04-30 16:46:57 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:46:57 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:46:57 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:46:57 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:46:59 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:47:00 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:47:01 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:47:02 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:47:04 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:47:05 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:47:06 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:47:07 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:47:08 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:47:10 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:47:10 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:47:10 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:47:10 [ 422 ] WARNING : ERROR Client failed! Return code: 210, stderr: Code: 210. DB::NetException: Connection refused (172.16.6.2:9000). (NETWORK_ERROR) (cluster.py:4008, wait_start) 2026-04-30 16:47:10 [ 422 ] ERROR : No time left to start. But process is still running. Will dump threads. (cluster.py:4013, wait_start) 2026-04-30 16:47:10 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:True cmd: ['bash', '-c', 'ps -C clickhouse'] (cluster.py:2173, exec_in_container) 2026-04-30 16:47:10 [ 422 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', 'ps -C clickhouse'] (cluster.py:113, run_and_check) 2026-04-30 16:47:10 [ 422 ] DEBUG : Stdout: PID TTY TIME CMD (cluster.py:121, run_and_check) 2026-04-30 16:47:10 [ 422 ] DEBUG : Stdout: 8378 ? 00:04:34 clickhouse (cluster.py:121, run_and_check) 2026-04-30 16:47:10 [ 422 ] INFO : PS RESULT: PID TTY TIME CMD 8378 ? 00:04:34 clickhouse (cluster.py:4019, wait_start) 2026-04-30 16:47:10 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:47:10 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:47:11 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:47:11 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "gdb -batch -ex 'thread apply all bt full' -p 8378"] (cluster.py:2173, exec_in_container) 2026-04-30 16:47:11 [ 422 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "gdb -batch -ex 'thread apply all bt full' -p 8378"] (cluster.py:113, run_and_check) 2026-04-30 16:47:12 [ 422 ] DEBUG : Stderr:warning: process 8378 is already traced by process 9185process 8378 is a zombie - the process has already terminated (cluster.py:123, run_and_check) 2026-04-30 16:47:12 [ 422 ] DEBUG : Stderr:ptrace: Operation not permitted. (cluster.py:123, run_and_check) 2026-04-30 16:47:12 [ 422 ] WARNING : Current start attempt failed. Will kill 8378 just in case. (cluster.py:3982, start_clickhouse) 2026-04-30 16:47:12 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:True cmd: ['bash', '-c', 'kill -9 8378'] (cluster.py:2173, exec_in_container) 2026-04-30 16:47:12 [ 422 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', 'kill -9 8378'] (cluster.py:113, run_and_check) ________________________________ test_send_data ________________________________ [gw0] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_send_data(started_cluster): update_configs(node_1_sleep_in_send_data=sleep_time) > check_query(expected_replica="node_2") test_hedged_requests/test.py:268: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ expected_replica = 'node_2', receive_timeout = 300 def check_query(expected_replica, receive_timeout=300): NODES["node"].restart_clickhouse() # Without hedged requests select query will last more than 30 seconds, # with hedged requests it will last just around 1-2 second start = time.time() result = NODES["node"].query( "SELECT hostName(), id FROM distributed ORDER BY id LIMIT 1 SETTINGS receive_timeout={}".format( receive_timeout ) ) query_time = time.time() - start > assert TSV(result) == TSV(expected_replica + "\t0") E AssertionError: assert node_1 0 == node_2 0 E + where node_1 0 = TSV('node_1\t0\n') E + and node_2 0 = TSV(('node_2' + '\t0')) test_hedged_requests/test.py:85: AssertionError ------------------------------ Captured log call ------------------------------- 2026-04-30 16:47:13 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1_1 detach:False nothrow:False cmd: ['bash', '-c', "echo '\n \n \n 0\n 30000\n 0\n \n \n' > /etc/clickhouse-server/users.d/users1.xml"] (cluster.py:2173, exec_in_container) 2026-04-30 16:47:13 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1_1', 'bash', '-c', "echo '\n \n \n 0\n 30000\n 0\n \n \n' > /etc/clickhouse-server/users.d/users1.xml"] (cluster.py:113, run_and_check) 2026-04-30 16:47:13 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_2_1 detach:False nothrow:False cmd: ['bash', '-c', "echo '\n \n \n 0\n 0\n 0\n \n \n' > /etc/clickhouse-server/users.d/users1.xml"] (cluster.py:2173, exec_in_container) 2026-04-30 16:47:13 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_2_1', 'bash', '-c', "echo '\n \n \n 0\n 0\n 0\n \n \n' > /etc/clickhouse-server/users.d/users1.xml"] (cluster.py:113, run_and_check) 2026-04-30 16:47:14 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_3_1 detach:False nothrow:False cmd: ['bash', '-c', "echo '\n \n \n 0\n 0\n 0\n \n \n' > /etc/clickhouse-server/users.d/users1.xml"] (cluster.py:2173, exec_in_container) 2026-04-30 16:47:14 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_3_1', 'bash', '-c', "echo '\n \n \n 0\n 0\n 0\n \n \n' > /etc/clickhouse-server/users.d/users1.xml"] (cluster.py:113, run_and_check) 2026-04-30 16:47:14 [ 422 ] DEBUG : Executing query SELECT value FROM system.settings WHERE name='sleep_in_send_tables_status_ms' on node_1 via HTTP interface (cluster.py:3850, http_query_and_get_answer_with_error) 2026-04-30 16:47:14 [ 422 ] DEBUG : Starting new HTTP connection (1): 172.16.6.4:8123 (connectionpool.py:245, _new_conn) 2026-04-30 16:47:14 [ 422 ] DEBUG : http://172.16.6.4:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_in_send_tables_status_ms%27 HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:47:15 [ 422 ] DEBUG : Executing query SELECT value FROM system.settings WHERE name='sleep_in_send_data_ms' on node_1 via HTTP interface (cluster.py:3850, http_query_and_get_answer_with_error) 2026-04-30 16:47:15 [ 422 ] DEBUG : Starting new HTTP connection (1): 172.16.6.4:8123 (connectionpool.py:245, _new_conn) 2026-04-30 16:47:15 [ 422 ] DEBUG : http://172.16.6.4:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_in_send_data_ms%27 HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:47:15 [ 422 ] DEBUG : Executing query SELECT value FROM system.settings WHERE name='sleep_after_receiving_query_ms' on node_1 via HTTP interface (cluster.py:3850, http_query_and_get_answer_with_error) 2026-04-30 16:47:15 [ 422 ] DEBUG : Starting new HTTP connection (1): 172.16.6.4:8123 (connectionpool.py:245, _new_conn) 2026-04-30 16:47:15 [ 422 ] DEBUG : http://172.16.6.4:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_after_receiving_query_ms%27 HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:47:15 [ 422 ] DEBUG : Executing query SELECT value FROM system.settings WHERE name='sleep_in_send_tables_status_ms' on node_1 via HTTP interface (cluster.py:3850, http_query_and_get_answer_with_error) 2026-04-30 16:47:15 [ 422 ] DEBUG : Starting new HTTP connection (1): 172.16.6.4:8123 (connectionpool.py:245, _new_conn) 2026-04-30 16:47:15 [ 422 ] DEBUG : http://172.16.6.4:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_in_send_tables_status_ms%27 HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:47:15 [ 422 ] DEBUG : Executing query SELECT value FROM system.settings WHERE name='sleep_in_send_data_ms' on node_1 via HTTP interface (cluster.py:3850, http_query_and_get_answer_with_error) 2026-04-30 16:47:15 [ 422 ] DEBUG : Starting new HTTP connection (1): 172.16.6.4:8123 (connectionpool.py:245, _new_conn) 2026-04-30 16:47:15 [ 422 ] DEBUG : http://172.16.6.4:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_in_send_data_ms%27 HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:47:15 [ 422 ] DEBUG : Executing query SELECT value FROM system.settings WHERE name='sleep_after_receiving_query_ms' on node_1 via HTTP interface (cluster.py:3850, http_query_and_get_answer_with_error) 2026-04-30 16:47:15 [ 422 ] DEBUG : Starting new HTTP connection (1): 172.16.6.4:8123 (connectionpool.py:245, _new_conn) 2026-04-30 16:47:15 [ 422 ] DEBUG : http://172.16.6.4:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_after_receiving_query_ms%27 HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:47:16 [ 422 ] DEBUG : Executing query SELECT value FROM system.settings WHERE name='sleep_in_send_tables_status_ms' on node_1 via HTTP interface (cluster.py:3850, http_query_and_get_answer_with_error) 2026-04-30 16:47:16 [ 422 ] DEBUG : Starting new HTTP connection (1): 172.16.6.4:8123 (connectionpool.py:245, _new_conn) 2026-04-30 16:47:16 [ 422 ] DEBUG : http://172.16.6.4:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_in_send_tables_status_ms%27 HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:47:16 [ 422 ] DEBUG : Executing query SELECT value FROM system.settings WHERE name='sleep_in_send_data_ms' on node_1 via HTTP interface (cluster.py:3850, http_query_and_get_answer_with_error) 2026-04-30 16:47:16 [ 422 ] DEBUG : Starting new HTTP connection (1): 172.16.6.4:8123 (connectionpool.py:245, _new_conn) 2026-04-30 16:47:16 [ 422 ] DEBUG : http://172.16.6.4:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_in_send_data_ms%27 HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:47:16 [ 422 ] DEBUG : Executing query SELECT value FROM system.settings WHERE name='sleep_after_receiving_query_ms' on node_1 via HTTP interface (cluster.py:3850, http_query_and_get_answer_with_error) 2026-04-30 16:47:16 [ 422 ] DEBUG : Starting new HTTP connection (1): 172.16.6.4:8123 (connectionpool.py:245, _new_conn) 2026-04-30 16:47:16 [ 422 ] DEBUG : http://172.16.6.4:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_after_receiving_query_ms%27 HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:47:16 [ 422 ] DEBUG : Executing query SELECT value FROM system.settings WHERE name='sleep_in_send_tables_status_ms' on node_2 via HTTP interface (cluster.py:3850, http_query_and_get_answer_with_error) 2026-04-30 16:47:16 [ 422 ] DEBUG : Starting new HTTP connection (1): 172.16.6.3:8123 (connectionpool.py:245, _new_conn) 2026-04-30 16:47:16 [ 422 ] DEBUG : http://172.16.6.3:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_in_send_tables_status_ms%27 HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:47:16 [ 422 ] DEBUG : Executing query SELECT value FROM system.settings WHERE name='sleep_in_send_data_ms' on node_2 via HTTP interface (cluster.py:3850, http_query_and_get_answer_with_error) 2026-04-30 16:47:16 [ 422 ] DEBUG : Starting new HTTP connection (1): 172.16.6.3:8123 (connectionpool.py:245, _new_conn) 2026-04-30 16:47:16 [ 422 ] DEBUG : http://172.16.6.3:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_in_send_data_ms%27 HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:47:16 [ 422 ] DEBUG : Executing query SELECT value FROM system.settings WHERE name='sleep_after_receiving_query_ms' on node_2 via HTTP interface (cluster.py:3850, http_query_and_get_answer_with_error) 2026-04-30 16:47:16 [ 422 ] DEBUG : Starting new HTTP connection (1): 172.16.6.3:8123 (connectionpool.py:245, _new_conn) 2026-04-30 16:47:17 [ 422 ] DEBUG : http://172.16.6.3:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_after_receiving_query_ms%27 HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:47:17 [ 422 ] DEBUG : Executing query SELECT value FROM system.settings WHERE name='sleep_in_send_tables_status_ms' on node_3 via HTTP interface (cluster.py:3850, http_query_and_get_answer_with_error) 2026-04-30 16:47:17 [ 422 ] DEBUG : Starting new HTTP connection (1): 172.16.6.5:8123 (connectionpool.py:245, _new_conn) 2026-04-30 16:47:17 [ 422 ] DEBUG : http://172.16.6.5:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_in_send_tables_status_ms%27 HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:47:17 [ 422 ] DEBUG : Executing query SELECT value FROM system.settings WHERE name='sleep_in_send_data_ms' on node_3 via HTTP interface (cluster.py:3850, http_query_and_get_answer_with_error) 2026-04-30 16:47:17 [ 422 ] DEBUG : Starting new HTTP connection (1): 172.16.6.5:8123 (connectionpool.py:245, _new_conn) 2026-04-30 16:47:17 [ 422 ] DEBUG : http://172.16.6.5:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_in_send_data_ms%27 HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:47:17 [ 422 ] DEBUG : Executing query SELECT value FROM system.settings WHERE name='sleep_after_receiving_query_ms' on node_3 via HTTP interface (cluster.py:3850, http_query_and_get_answer_with_error) 2026-04-30 16:47:17 [ 422 ] DEBUG : Starting new HTTP connection (1): 172.16.6.5:8123 (connectionpool.py:245, _new_conn) 2026-04-30 16:47:17 [ 422 ] DEBUG : http://172.16.6.5:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_after_receiving_query_ms%27 HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:47:17 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:True cmd: ['bash', '-c', 'ps -C clickhouse'] (cluster.py:2173, exec_in_container) 2026-04-30 16:47:17 [ 422 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', 'ps -C clickhouse'] (cluster.py:113, run_and_check) 2026-04-30 16:47:18 [ 422 ] DEBUG : Stdout: PID TTY TIME CMD (cluster.py:121, run_and_check) 2026-04-30 16:47:18 [ 422 ] DEBUG : Stdout: 8378 ? 00:04:34 clickhouse (cluster.py:121, run_and_check) 2026-04-30 16:47:18 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', 'pkill clickhouse'] (cluster.py:2173, exec_in_container) 2026-04-30 16:47:18 [ 422 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', 'pkill clickhouse'] (cluster.py:113, run_and_check) 2026-04-30 16:47:18 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:47:18 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:47:19 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:47:20 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:47:20 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:47:20 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:47:21 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:47:21 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:47:22 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:47:23 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:47:23 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:47:24 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:47:25 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:47:25 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:47:25 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:47:26 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:47:26 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:47:27 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:47:28 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:47:28 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:47:28 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:47:29 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:47:29 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:47:30 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:47:31 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:47:31 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:47:31 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:47:32 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:47:32 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:47:33 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:47:34 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:47:34 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:47:34 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:47:35 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:47:35 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:47:36 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:47:37 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:47:37 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:47:37 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:47:38 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:47:38 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:47:39 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:47:40 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:47:40 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:47:40 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:47:41 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:47:41 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:47:42 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:47:43 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:47:43 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:47:43 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:47:44 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:47:44 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:47:45 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:47:46 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:47:46 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:47:46 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:47:47 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:47:47 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:47:48 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:47:49 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:47:49 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:47:49 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:47:50 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:47:50 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:47:51 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:47:52 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:47:52 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:47:52 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:47:53 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:47:53 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:47:54 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:47:55 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:47:55 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:47:55 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:47:56 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:47:56 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:47:57 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:47:58 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:47:58 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:47:58 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:47:59 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:47:59 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:47:59 [ 422 ] DEBUG : Stdout:8378 (cluster.py:121, run_and_check) 2026-04-30 16:48:00 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:48:00 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:48:01 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:48:01 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:48:02 [ 422 ] DEBUG : No clickhouse process running. Start new one. (cluster.py:3964, start_clickhouse) 2026-04-30 16:48:02 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', 'clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon'] (cluster.py:2173, exec_in_container) 2026-04-30 16:48:02 [ 422 ] DEBUG : Command:['docker', 'exec', '-u', '0', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', 'clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon'] (cluster.py:113, run_and_check) 2026-04-30 16:48:04 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:48:04 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:48:05 [ 422 ] DEBUG : Stdout:10265 (cluster.py:121, run_and_check) 2026-04-30 16:48:05 [ 422 ] DEBUG : Clickhouse process running. (cluster.py:3975, start_clickhouse) 2026-04-30 16:48:05 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:48:05 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:48:06 [ 422 ] DEBUG : Stdout:10265 (cluster.py:121, run_and_check) 2026-04-30 16:48:06 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:48:07 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:48:08 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:48:09 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:48:11 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:48:12 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:48:13 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:48:14 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:48:15 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:48:16 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:48:18 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:48:18 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:48:18 [ 422 ] DEBUG : Stdout:10265 (cluster.py:121, run_and_check) 2026-04-30 16:48:18 [ 422 ] WARNING : ERROR Client failed! Return code: 210, stderr: Code: 210. DB::NetException: Connection refused (172.16.6.2:9000). (NETWORK_ERROR) (cluster.py:4008, wait_start) 2026-04-30 16:48:18 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:48:18 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:48:19 [ 422 ] DEBUG : Stdout:10265 (cluster.py:121, run_and_check) 2026-04-30 16:48:19 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:48:20 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:48:21 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:48:22 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:48:23 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:48:24 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:48:26 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:48:27 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:48:28 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:48:29 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:48:30 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:48:30 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:48:31 [ 422 ] DEBUG : Stdout:10265 (cluster.py:121, run_and_check) 2026-04-30 16:48:31 [ 422 ] WARNING : ERROR Client failed! Return code: 210, stderr: Code: 210. DB::NetException: Connection refused (172.16.6.2:9000). (NETWORK_ERROR) (cluster.py:4008, wait_start) 2026-04-30 16:48:31 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:48:31 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:48:32 [ 422 ] DEBUG : Stdout:10265 (cluster.py:121, run_and_check) 2026-04-30 16:48:32 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:48:33 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:48:34 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:48:35 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:48:36 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:48:37 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:48:38 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:48:39 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:48:41 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:48:42 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:48:43 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:48:43 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:48:44 [ 422 ] DEBUG : Stdout:10265 (cluster.py:121, run_and_check) 2026-04-30 16:48:44 [ 422 ] WARNING : ERROR Client failed! Return code: 210, stderr: Code: 210. DB::NetException: Connection refused (172.16.6.2:9000). (NETWORK_ERROR) (cluster.py:4008, wait_start) 2026-04-30 16:48:44 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:48:44 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:48:45 [ 422 ] DEBUG : Stdout:10265 (cluster.py:121, run_and_check) 2026-04-30 16:48:45 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:48:46 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:48:47 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:48:48 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:48:49 [ 422 ] DEBUG : Executing query SELECT hostName(), id FROM distributed ORDER BY id LIMIT 1 SETTINGS receive_timeout=300 on node (cluster.py:3602, query) _______________________________ test_send_data2 ________________________________ [gw0] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_send_data2(started_cluster): update_configs( node_1_sleep_in_send_data=sleep_time, node_2_sleep_in_send_data=sleep_time ) > check_query(expected_replica="node_3") test_hedged_requests/test.py:276: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ expected_replica = 'node_3', receive_timeout = 300 def check_query(expected_replica, receive_timeout=300): NODES["node"].restart_clickhouse() # Without hedged requests select query will last more than 30 seconds, # with hedged requests it will last just around 1-2 second start = time.time() result = NODES["node"].query( "SELECT hostName(), id FROM distributed ORDER BY id LIMIT 1 SETTINGS receive_timeout={}".format( receive_timeout ) ) query_time = time.time() - start > assert TSV(result) == TSV(expected_replica + "\t0") E AssertionError: assert node_2 0 == node_3 0 E + where node_2 0 = TSV('node_2\t0\n') E + and node_3 0 = TSV(('node_3' + '\t0')) test_hedged_requests/test.py:85: AssertionError ------------------------------ Captured log call ------------------------------- 2026-04-30 16:49:21 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1_1 detach:False nothrow:False cmd: ['bash', '-c', "echo '\n \n \n 0\n 30000\n 0\n \n \n' > /etc/clickhouse-server/users.d/users1.xml"] (cluster.py:2173, exec_in_container) 2026-04-30 16:49:21 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1_1', 'bash', '-c', "echo '\n \n \n 0\n 30000\n 0\n \n \n' > /etc/clickhouse-server/users.d/users1.xml"] (cluster.py:113, run_and_check) 2026-04-30 16:49:22 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_2_1 detach:False nothrow:False cmd: ['bash', '-c', "echo '\n \n \n 0\n 30000\n 0\n \n \n' > /etc/clickhouse-server/users.d/users1.xml"] (cluster.py:2173, exec_in_container) 2026-04-30 16:49:22 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_2_1', 'bash', '-c', "echo '\n \n \n 0\n 30000\n 0\n \n \n' > /etc/clickhouse-server/users.d/users1.xml"] (cluster.py:113, run_and_check) 2026-04-30 16:49:22 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_3_1 detach:False nothrow:False cmd: ['bash', '-c', "echo '\n \n \n 0\n 0\n 0\n \n \n' > /etc/clickhouse-server/users.d/users1.xml"] (cluster.py:2173, exec_in_container) 2026-04-30 16:49:22 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_3_1', 'bash', '-c', "echo '\n \n \n 0\n 0\n 0\n \n \n' > /etc/clickhouse-server/users.d/users1.xml"] (cluster.py:113, run_and_check) 2026-04-30 16:49:23 [ 422 ] DEBUG : Executing query SELECT value FROM system.settings WHERE name='sleep_in_send_tables_status_ms' on node_1 via HTTP interface (cluster.py:3850, http_query_and_get_answer_with_error) 2026-04-30 16:49:23 [ 422 ] DEBUG : Starting new HTTP connection (1): 172.16.6.4:8123 (connectionpool.py:245, _new_conn) 2026-04-30 16:49:23 [ 422 ] DEBUG : http://172.16.6.4:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_in_send_tables_status_ms%27 HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:49:23 [ 422 ] DEBUG : Executing query SELECT value FROM system.settings WHERE name='sleep_in_send_data_ms' on node_1 via HTTP interface (cluster.py:3850, http_query_and_get_answer_with_error) 2026-04-30 16:49:23 [ 422 ] DEBUG : Starting new HTTP connection (1): 172.16.6.4:8123 (connectionpool.py:245, _new_conn) 2026-04-30 16:49:23 [ 422 ] DEBUG : http://172.16.6.4:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_in_send_data_ms%27 HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:49:23 [ 422 ] DEBUG : Executing query SELECT value FROM system.settings WHERE name='sleep_after_receiving_query_ms' on node_1 via HTTP interface (cluster.py:3850, http_query_and_get_answer_with_error) 2026-04-30 16:49:23 [ 422 ] DEBUG : Starting new HTTP connection (1): 172.16.6.4:8123 (connectionpool.py:245, _new_conn) 2026-04-30 16:49:24 [ 422 ] DEBUG : http://172.16.6.4:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_after_receiving_query_ms%27 HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:49:24 [ 422 ] DEBUG : Executing query SELECT value FROM system.settings WHERE name='sleep_in_send_tables_status_ms' on node_2 via HTTP interface (cluster.py:3850, http_query_and_get_answer_with_error) 2026-04-30 16:49:24 [ 422 ] DEBUG : Starting new HTTP connection (1): 172.16.6.3:8123 (connectionpool.py:245, _new_conn) 2026-04-30 16:49:24 [ 422 ] DEBUG : http://172.16.6.3:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_in_send_tables_status_ms%27 HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:49:24 [ 422 ] DEBUG : Executing query SELECT value FROM system.settings WHERE name='sleep_in_send_data_ms' on node_2 via HTTP interface (cluster.py:3850, http_query_and_get_answer_with_error) 2026-04-30 16:49:24 [ 422 ] DEBUG : Starting new HTTP connection (1): 172.16.6.3:8123 (connectionpool.py:245, _new_conn) 2026-04-30 16:49:24 [ 422 ] DEBUG : http://172.16.6.3:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_in_send_data_ms%27 HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:49:24 [ 422 ] DEBUG : Executing query SELECT value FROM system.settings WHERE name='sleep_after_receiving_query_ms' on node_2 via HTTP interface (cluster.py:3850, http_query_and_get_answer_with_error) 2026-04-30 16:49:24 [ 422 ] DEBUG : Starting new HTTP connection (1): 172.16.6.3:8123 (connectionpool.py:245, _new_conn) 2026-04-30 16:49:24 [ 422 ] DEBUG : http://172.16.6.3:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_after_receiving_query_ms%27 HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:49:24 [ 422 ] DEBUG : Executing query SELECT value FROM system.settings WHERE name='sleep_in_send_tables_status_ms' on node_3 via HTTP interface (cluster.py:3850, http_query_and_get_answer_with_error) 2026-04-30 16:49:24 [ 422 ] DEBUG : Starting new HTTP connection (1): 172.16.6.5:8123 (connectionpool.py:245, _new_conn) 2026-04-30 16:49:24 [ 422 ] DEBUG : http://172.16.6.5:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_in_send_tables_status_ms%27 HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:49:24 [ 422 ] DEBUG : Executing query SELECT value FROM system.settings WHERE name='sleep_in_send_data_ms' on node_3 via HTTP interface (cluster.py:3850, http_query_and_get_answer_with_error) 2026-04-30 16:49:24 [ 422 ] DEBUG : Starting new HTTP connection (1): 172.16.6.5:8123 (connectionpool.py:245, _new_conn) 2026-04-30 16:49:24 [ 422 ] DEBUG : http://172.16.6.5:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_in_send_data_ms%27 HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:49:24 [ 422 ] DEBUG : Executing query SELECT value FROM system.settings WHERE name='sleep_after_receiving_query_ms' on node_3 via HTTP interface (cluster.py:3850, http_query_and_get_answer_with_error) 2026-04-30 16:49:24 [ 422 ] DEBUG : Starting new HTTP connection (1): 172.16.6.5:8123 (connectionpool.py:245, _new_conn) 2026-04-30 16:49:25 [ 422 ] DEBUG : http://172.16.6.5:8123 "GET /?query=SELECT+value+FROM+system.settings+WHERE+name%3D%27sleep_after_receiving_query_ms%27 HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-04-30 16:49:25 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:True cmd: ['bash', '-c', 'ps -C clickhouse'] (cluster.py:2173, exec_in_container) 2026-04-30 16:49:25 [ 422 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', 'ps -C clickhouse'] (cluster.py:113, run_and_check) 2026-04-30 16:49:25 [ 422 ] DEBUG : Stdout: PID TTY TIME CMD (cluster.py:121, run_and_check) 2026-04-30 16:49:25 [ 422 ] DEBUG : Stdout: 10265 ? 00:04:29 clickhouse (cluster.py:121, run_and_check) 2026-04-30 16:49:25 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', 'pkill clickhouse'] (cluster.py:2173, exec_in_container) 2026-04-30 16:49:25 [ 422 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', 'pkill clickhouse'] (cluster.py:113, run_and_check) 2026-04-30 16:49:26 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:49:26 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:49:26 [ 422 ] DEBUG : Stdout:10265 (cluster.py:121, run_and_check) 2026-04-30 16:49:27 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:49:27 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:49:28 [ 422 ] DEBUG : Stdout:10265 (cluster.py:121, run_and_check) 2026-04-30 16:49:29 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:49:29 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:49:29 [ 422 ] DEBUG : Stdout:10265 (cluster.py:121, run_and_check) 2026-04-30 16:49:30 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:49:30 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:49:31 [ 422 ] DEBUG : Stdout:10265 (cluster.py:121, run_and_check) 2026-04-30 16:49:32 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:49:32 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:49:32 [ 422 ] DEBUG : Stdout:10265 (cluster.py:121, run_and_check) 2026-04-30 16:49:33 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:49:33 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:49:34 [ 422 ] DEBUG : Stdout:10265 (cluster.py:121, run_and_check) 2026-04-30 16:49:34 [ 422 ] DEBUG : Stdout:11126 (cluster.py:121, run_and_check) 2026-04-30 16:49:35 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:49:35 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:49:35 [ 422 ] DEBUG : Stdout:10265 (cluster.py:121, run_and_check) 2026-04-30 16:49:36 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:49:36 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:49:37 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:49:37 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:49:37 [ 422 ] DEBUG : No clickhouse process running. Start new one. (cluster.py:3964, start_clickhouse) 2026-04-30 16:49:37 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', 'clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon'] (cluster.py:2173, exec_in_container) 2026-04-30 16:49:37 [ 422 ] DEBUG : Command:['docker', 'exec', '-u', '0', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', 'clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon'] (cluster.py:113, run_and_check) 2026-04-30 16:49:39 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:49:39 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:49:40 [ 422 ] DEBUG : Stdout:11168 (cluster.py:121, run_and_check) 2026-04-30 16:49:40 [ 422 ] DEBUG : Clickhouse process running. (cluster.py:3975, start_clickhouse) 2026-04-30 16:49:40 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:49:40 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:49:40 [ 422 ] DEBUG : Stdout:11168 (cluster.py:121, run_and_check) 2026-04-30 16:49:40 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:49:41 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:49:43 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:49:44 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:49:45 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:49:46 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:49:47 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:49:48 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:49:49 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:49:50 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:49:51 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:49:51 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:49:52 [ 422 ] DEBUG : Stdout:11168 (cluster.py:121, run_and_check) 2026-04-30 16:49:52 [ 422 ] WARNING : ERROR Client failed! Return code: 210, stderr: Code: 210. DB::NetException: Connection refused (172.16.6.2:9000). (NETWORK_ERROR) (cluster.py:4008, wait_start) 2026-04-30 16:49:52 [ 422 ] DEBUG : run container_id:roottesthedgedrequests_gw0_node_1 detach:False nothrow:False cmd: ['bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:2173, exec_in_container) 2026-04-30 16:49:52 [ 422 ] DEBUG : Command:['docker', 'exec', 'roottesthedgedrequests_gw0_node_1', 'bash', '-c', "ps ax | grep 'clickhouse' | grep -v 'grep' | grep -v 'coproc' | grep -v 'bash -c' | awk '{print $1}'"] (cluster.py:113, run_and_check) 2026-04-30 16:49:52 [ 422 ] DEBUG : Stdout:11168 (cluster.py:121, run_and_check) 2026-04-30 16:49:52 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:49:53 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:49:54 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:49:56 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:49:57 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:49:58 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:49:59 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:50:00 [ 422 ] DEBUG : Executing query select 20 on node (cluster.py:3602, query) 2026-04-30 16:50:01 [ 422 ] DEBUG : Executing query SELECT hostName(), id FROM distributed ORDER BY id LIMIT 1 SETTINGS receive_timeout=300 on node (cluster.py:3602, query) ============================== slowest durations =============================== 599.86s call test_globs_in_filepath/test.py::test_deep_structure 554.90s call test_distributed_load_balancing/test.py::test_load_balancing_first_or_random 407.06s call test_hedged_requests/test.py::test_receive_timeout1 398.44s call test_distributed_load_balancing/test.py::test_load_balancing_hostname_levenshtein_distance 316.47s call test_distributed_load_balancing/test.py::test_load_balancing_default 298.02s call test_distributed_load_balancing/test.py::test_load_balancing_nearest_hostname 286.81s call test_distributed_load_balancing/test.py::test_load_balancing_in_order 167.60s setup test_endpoint_macro_substitution/test.py::test_different_types 138.82s call test_http_handlers_config/test.py::test_defaults_http_handlers_config_order 134.45s call test_hedged_requests/test.py::test_receive_timeout2 128.32s call test_hedged_requests/test.py::test_send_data 118.11s setup test_encrypted_disk_replication/test.py::test_replication 95.59s call test_http_handlers_config/test.py::test_headers_in_response 91.90s call test_hedged_requests/test.py::test_long_query 88.90s call test_http_handlers_config/test.py::test_relative_path_static_handler 88.14s call test_http_handlers_config/test.py::test_predefined_query_handler 87.26s call test_http_handlers_config/test.py::test_prometheus_handler 86.92s call test_http_handlers_config/test.py::test_dynamic_query_handler 85.89s call test_http_handlers_config/test.py::test_fixed_static_handler 84.79s call test_http_handlers_config/test.py::test_replicas_status_handler 84.64s setup test_distributed_inter_server_secret/test.py::test_insecure 75.01s setup test_distributed_load_balancing/test.py::test_distributed_replica_max_ignored_errors 74.95s call test_http_handlers_config/test.py::test_defaults_http_handlers 73.72s setup test_executable_user_defined_functions_config_reload/test.py::test 72.90s call test_hedged_requests/test.py::test_send_data2 70.21s call test_http_handlers_config/test.py::test_absolute_path_static_handler 69.71s call test_hedged_requests/test.py::test_initial_receive_timeout 68.91s call test_hedged_requests/test.py::test_async_query_sending 65.35s call test_http_handlers_config/test.py::test_config_static_handler 65.25s setup test_hedged_requests/test.py::test_async_connect 61.89s setup test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_corrupted_big[0] 59.19s setup test_hot_reload_storage_policy/test.py::test_hot_reload_policy 58.98s setup test_distributed_insert_backward_compatibility/test.py::test_distributed_in_tuple 57.73s setup test_graphite_merge_tree/test.py::test_broken_partial_rollup 57.58s setup test_distributed_format/test.py::test_single_file[test_cluster_internal_replication] 56.81s call test_hedged_requests/test.py::test_stuck_replica 55.94s setup test_globs_in_filepath/test.py::test_deep_structure 55.78s setup test_grpc_protocol_ssl/test.py::test_insecure_channel 55.33s call test_hedged_requests/test.py::test_combination4 50.80s call test_hedged_requests/test.py::test_send_table_status_sleep 50.77s call test_hedged_requests/test.py::test_send_table_status_sleep2 50.47s setup test_dotnet_client/test.py::test_dotnet_client 50.37s call test_hedged_requests/test.py::test_combination3 48.40s setup test_input_format_parallel_parsing_memory_tracking/test.py::test_memory_tracking_total 46.28s call test_globs_in_filepath/test.py::test_linear_structure 44.97s call test_hedged_requests/test.py::test_async_connect 43.95s call test_distributed_insert_backward_compatibility/test.py::test_distributed_in_tuple 42.65s call test_hedged_requests/test.py::test_combination2 41.33s call test_graphite_merge_tree/test.py::test_path_dangling_pointer 39.52s call test_hedged_requests/test.py::test_combination1 33.36s call test_hot_reload_storage_policy/test.py::test_hot_reload_policy_distributed_table 33.02s teardown test_encrypted_disk_replication/test.py::test_zero_copy_replication 32.61s call test_distributed_load_balancing/test.py::test_load_balancing_priority_round_robin[dist_priority_negative] 31.50s teardown test_endpoint_macro_substitution/test.py::test_select_by_type 29.37s call test_distributed_load_balancing/test.py::test_load_balancing_priority_round_robin[dist_priority] 27.24s call test_distributed_load_balancing/test.py::test_load_balancing_round_robin 26.54s call test_executable_user_defined_functions_config_reload/test.py::test 25.10s teardown test_globs_in_filepath/test.py::test_table_function_and_virtual_columns 24.09s teardown test_hedged_requests/test.py::test_stuck_replica 21.98s call test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_corrupted_big[0] 20.01s call test_input_format_parallel_parsing_memory_tracking/test.py::test_memory_tracking_total 19.80s call test_distributed_load_balancing/test.py::test_distributed_replica_max_ignored_errors 19.30s teardown test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_corrupted_big[0] 18.41s teardown test_distributed_inter_server_secret/test.py::test_user_secure_cluster[pass-foo] 17.77s teardown test_graphite_merge_tree/test.py::test_wrong_rollup_config 17.23s call test_globs_in_filepath/test.py::test_strange_filenames 17.15s teardown test_distributed_insert_backward_compatibility/test.py::test_distributed_in_tuple 16.75s call test_globs_in_filepath/test.py::test_table_function_and_virtual_columns 15.30s teardown test_dotnet_client/test.py::test_dotnet_client 15.04s call test_dotnet_client/test.py::test_dotnet_client 14.74s call test_distributed_inter_server_secret/test.py::test_secure_insert_buffer_async 13.66s teardown test_grpc_protocol_ssl/test.py::test_wrong_client_certificate 12.33s teardown test_distributed_load_balancing/test.py::test_load_balancing_round_robin 11.35s teardown test_input_format_parallel_parsing_memory_tracking/test.py::test_memory_tracking_total 11.29s call test_distributed_inter_server_secret/test.py::test_secure_cluster_distributed_over_distributed_different_users_cluster 10.32s call test_hot_reload_storage_policy/test.py::test_hot_reload_policy 9.63s call test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[default-] 9.03s teardown test_hot_reload_storage_policy/test.py::test_hot_reload_policy_distributed_table 8.62s call test_encrypted_disk_replication/test.py::test_replication 8.12s call test_distributed_inter_server_secret/test.py::test_user_secure_cluster[pass-foo] 8.10s call test_distributed_inter_server_secret/test.py::test_user_secure_cluster[nopass-] 7.86s call test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[pass-foo] 7.84s call test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[nopass-] 7.83s call test_distributed_inter_server_secret/test.py::test_user_secure_cluster[default-] 7.79s teardown test_distributed_format/test.py::test_two_files[test_cluster_no_internal_replication] 7.60s teardown test_executable_user_defined_functions_config_reload/test.py::test 6.68s call test_distributed_format/test.py::test_two_files[test_cluster_internal_replication] 6.39s call test_distributed_format/test.py::test_single_file_old[test_cluster_no_internal_replication] 5.97s call test_distributed_format/test.py::test_single_file[test_cluster_no_internal_replication] 5.88s call test_distributed_format/test.py::test_two_files[test_cluster_no_internal_replication] 5.80s call test_encrypted_disk_replication/test.py::test_zero_copy_replication 5.54s call test_distributed_inter_server_secret/test.py::test_secure_insert_async 5.51s call test_distributed_format/test.py::test_single_file[test_cluster_internal_replication] 5.34s call test_distributed_format/test.py::test_single_file_old[test_cluster_internal_replication] 5.11s call test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_insecure_cluster[nopass-] 5.10s call test_distributed_inter_server_secret/test.py::test_secure_disagree_insert 4.69s call test_distributed_inter_server_secret/test.py::test_insecure_insert_async 4.58s setup test_distributed_load_balancing/test.py::test_load_balancing_in_order 4.50s call test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_insecure_cluster[pass-foo] 4.34s call test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_insecure_cluster[nopass-] 4.32s call test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_insecure_cluster[default-] 4.32s call test_graphite_merge_tree/test.py::test_combined_rules 4.06s call test_distributed_inter_server_secret/test.py::test_secure_insert_sync 4.03s call test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_secure_cluster[pass-foo] 3.96s call test_graphite_merge_tree/test.py::test_combined_rules_with_default 3.83s setup test_distributed_load_balancing/test.py::test_load_balancing_first_or_random 3.80s call test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_insecure_cluster[default-] 3.79s call test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_insecure_cluster[pass-foo] 3.78s call test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_secure_cluster[nopass-] 3.74s call test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_secure_cluster[default-] 3.69s call test_distributed_inter_server_secret/test.py::test_secure_cluster_distributed_over_distributed_different_users_remote 3.66s call test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[pass-foo] 3.58s call test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[nopass-] 3.55s call test_endpoint_macro_substitution/test.py::test_select_by_type 3.43s setup test_distributed_load_balancing/test.py::test_load_balancing_hostname_levenshtein_distance 3.39s call test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[default-] 3.36s call test_graphite_merge_tree/test.py::test_multiple_output_blocks 3.21s call test_distributed_inter_server_secret/test.py::test_insecure_insert_sync 2.96s call test_graphite_merge_tree/test.py::test_rollup_versions 2.52s setup test_distributed_load_balancing/test.py::test_load_balancing_round_robin 2.40s call test_graphite_merge_tree/test.py::test_broken_partial_rollup 2.33s call test_graphite_merge_tree/test.py::test_system_graphite_retentions 2.29s setup test_distributed_load_balancing/test.py::test_load_balancing_default 2.23s setup test_distributed_load_balancing/test.py::test_load_balancing_priority_round_robin[dist_priority] 2.19s setup test_distributed_load_balancing/test.py::test_load_balancing_nearest_hostname 2.17s call test_graphite_merge_tree/test.py::test_rollup_aggregation 2.16s call test_grpc_protocol_ssl/test.py::test_insecure_channel 2.13s setup test_distributed_load_balancing/test.py::test_load_balancing_priority_round_robin[dist_priority_negative] 2.04s call test_graphite_merge_tree/test.py::test_paths_not_matching_any_pattern 2.00s call test_grpc_protocol_ssl/test.py::test_wrong_client_certificate 1.81s teardown test_encrypted_disk_replication/test.py::test_replication 1.26s call test_endpoint_macro_substitution/test.py::test_different_types 1.20s call test_graphite_merge_tree/test.py::test_wrong_rollup_config 1.13s call test_distributed_inter_server_secret/test.py::test_secure 1.04s call test_graphite_merge_tree/test.py::test_multiple_paths_and_versions 1.03s call test_distributed_inter_server_secret/test.py::test_secure_disagree 1.00s teardown test_graphite_merge_tree/test.py::test_combined_rules 0.98s setup test_graphite_merge_tree/test.py::test_path_dangling_pointer 0.96s call test_distributed_inter_server_secret/test.py::test_insecure 0.92s setup test_graphite_merge_tree/test.py::test_paths_not_matching_any_pattern 0.91s teardown test_graphite_merge_tree/test.py::test_system_graphite_retentions 0.90s setup test_graphite_merge_tree/test.py::test_multiple_paths_and_versions 0.84s call test_graphite_merge_tree/test.py::test_rollup_aggregation_2 0.82s setup test_graphite_merge_tree/test.py::test_rollup_versions 0.81s teardown test_graphite_merge_tree/test.py::test_multiple_output_blocks 0.81s setup test_graphite_merge_tree/test.py::test_wrong_rollup_config 0.78s setup test_graphite_merge_tree/test.py::test_system_graphite_retentions 0.76s setup test_graphite_merge_tree/test.py::test_combined_rules_with_default 0.76s teardown test_graphite_merge_tree/test.py::test_broken_partial_rollup 0.76s setup test_graphite_merge_tree/test.py::test_rollup_aggregation 0.76s teardown test_graphite_merge_tree/test.py::test_path_dangling_pointer 0.71s setup test_graphite_merge_tree/test.py::test_combined_rules 0.70s setup test_graphite_merge_tree/test.py::test_multiple_output_blocks 0.68s teardown test_graphite_merge_tree/test.py::test_paths_not_matching_any_pattern 0.68s teardown test_graphite_merge_tree/test.py::test_rollup_aggregation 0.66s teardown test_graphite_merge_tree/test.py::test_multiple_paths_and_versions 0.66s setup test_graphite_merge_tree/test.py::test_rollup_aggregation_2 0.66s teardown test_graphite_merge_tree/test.py::test_rollup_versions 0.63s teardown test_graphite_merge_tree/test.py::test_rollup_aggregation_2 0.59s teardown test_graphite_merge_tree/test.py::test_combined_rules_with_default 0.37s call test_grpc_protocol_ssl/test.py::test_secure_channel 0.20s setup test_http_handlers_config/test.py::test_absolute_path_static_handler 0.04s teardown test_endpoint_macro_substitution/test.py::test_different_types 0.02s setup test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[nopass-] 0.02s teardown test_grpc_protocol_ssl/test.py::test_secure_channel 0.02s setup test_distributed_format/test.py::test_single_file[test_cluster_no_internal_replication] 0.02s setup test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_secure_cluster[pass-foo] 0.02s teardown test_distributed_inter_server_secret/test.py::test_user_secure_cluster[nopass-] 0.01s setup test_distributed_inter_server_secret/test.py::test_secure_insert_async 0.01s setup test_globs_in_filepath/test.py::test_table_function_and_virtual_columns 0.01s setup test_distributed_inter_server_secret/test.py::test_secure_cluster_distributed_over_distributed_different_users_remote 0.01s setup test_http_handlers_config/test.py::test_dynamic_query_handler 0.01s teardown test_distributed_format/test.py::test_single_file_old[test_cluster_internal_replication] 0.01s setup test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[pass-foo] 0.01s setup test_hedged_requests/test.py::test_receive_timeout1 0.01s teardown test_hedged_requests/test.py::test_initial_receive_timeout 0.01s teardown test_http_handlers_config/test.py::test_replicas_status_handler 0.01s teardown test_globs_in_filepath/test.py::test_deep_structure 0.01s teardown test_hedged_requests/test.py::test_async_connect 0.01s teardown test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_secure_cluster[nopass-] 0.01s teardown test_distributed_inter_server_secret/test.py::test_secure_cluster_distributed_over_distributed_different_users_remote 0.01s setup test_http_handlers_config/test.py::test_defaults_http_handlers 0.01s teardown test_distributed_load_balancing/test.py::test_load_balancing_first_or_random 0.01s setup test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_insecure_cluster[nopass-] 0.01s teardown test_distributed_load_balancing/test.py::test_load_balancing_priority_round_robin[dist_priority_negative] 0.01s setup test_distributed_inter_server_secret/test.py::test_user_secure_cluster[default-] 0.01s teardown test_http_handlers_config/test.py::test_fixed_static_handler 0.01s setup test_distributed_inter_server_secret/test.py::test_user_secure_cluster[nopass-] 0.01s setup test_distributed_inter_server_secret/test.py::test_secure_disagree_insert 0.01s setup test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[default-] 0.01s setup test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_insecure_cluster[default-] 0.01s teardown test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[pass-foo] 0.01s teardown test_globs_in_filepath/test.py::test_strange_filenames 0.01s teardown test_distributed_format/test.py::test_single_file[test_cluster_internal_replication] 0.01s setup test_http_handlers_config/test.py::test_relative_path_static_handler 0.01s teardown test_distributed_load_balancing/test.py::test_load_balancing_in_order 0.01s teardown test_http_handlers_config/test.py::test_prometheus_handler 0.01s teardown test_distributed_format/test.py::test_two_files[test_cluster_internal_replication] 0.01s setup test_distributed_inter_server_secret/test.py::test_user_secure_cluster[pass-foo] 0.01s teardown test_http_handlers_config/test.py::test_defaults_http_handlers_config_order 0.01s setup test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[default-] 0.01s setup test_hedged_requests/test.py::test_long_query 0.00s teardown test_http_handlers_config/test.py::test_absolute_path_static_handler 0.00s teardown test_distributed_format/test.py::test_single_file[test_cluster_no_internal_replication] 0.00s teardown test_hedged_requests/test.py::test_combination1 0.00s teardown test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[pass-foo] 0.00s setup test_hedged_requests/test.py::test_initial_receive_timeout 0.00s teardown test_http_handlers_config/test.py::test_defaults_http_handlers 0.00s setup test_distributed_inter_server_secret/test.py::test_secure_insert_sync 0.00s teardown test_distributed_load_balancing/test.py::test_load_balancing_default 0.00s setup test_hedged_requests/test.py::test_combination2 0.00s setup test_distributed_format/test.py::test_single_file_old[test_cluster_internal_replication] 0.00s setup test_distributed_inter_server_secret/test.py::test_insecure_insert_async 0.00s setup test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_insecure_cluster[default-] 0.00s teardown test_http_handlers_config/test.py::test_relative_path_static_handler 0.00s setup test_hot_reload_storage_policy/test.py::test_hot_reload_policy_distributed_table 0.00s teardown test_http_handlers_config/test.py::test_dynamic_query_handler 0.00s teardown test_distributed_inter_server_secret/test.py::test_insecure 0.00s teardown test_distributed_inter_server_secret/test.py::test_secure_disagree_insert 0.00s teardown test_distributed_load_balancing/test.py::test_load_balancing_nearest_hostname 0.00s setup test_http_handlers_config/test.py::test_defaults_http_handlers_config_order 0.00s setup test_hedged_requests/test.py::test_async_query_sending 0.00s teardown test_distributed_inter_server_secret/test.py::test_secure_insert_sync 0.00s setup test_globs_in_filepath/test.py::test_linear_structure 0.00s teardown test_distributed_load_balancing/test.py::test_load_balancing_priority_round_robin[dist_priority] 0.00s teardown test_distributed_inter_server_secret/test.py::test_insecure_insert_sync 0.00s teardown test_http_handlers_config/test.py::test_headers_in_response 0.00s setup test_hedged_requests/test.py::test_combination1 0.00s setup test_http_handlers_config/test.py::test_headers_in_response 0.00s teardown test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_secure_cluster[pass-foo] 0.00s setup test_distributed_inter_server_secret/test.py::test_secure 0.00s setup test_globs_in_filepath/test.py::test_strange_filenames 0.00s setup test_distributed_format/test.py::test_single_file_old[test_cluster_no_internal_replication] 0.00s teardown test_hedged_requests/test.py::test_send_data 0.00s teardown test_hedged_requests/test.py::test_receive_timeout1 0.00s teardown test_hot_reload_storage_policy/test.py::test_hot_reload_policy 0.00s teardown test_hedged_requests/test.py::test_combination3 0.00s setup test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_secure_cluster[nopass-] 0.00s teardown test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_insecure_cluster[nopass-] 0.00s setup test_http_handlers_config/test.py::test_fixed_static_handler 0.00s teardown test_hedged_requests/test.py::test_async_query_sending 0.00s setup test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_insecure_cluster[pass-foo] 0.00s setup test_distributed_format/test.py::test_two_files[test_cluster_no_internal_replication] 0.00s teardown test_http_handlers_config/test.py::test_predefined_query_handler 0.00s teardown test_hedged_requests/test.py::test_send_data2 0.00s teardown test_globs_in_filepath/test.py::test_linear_structure 0.00s setup test_grpc_protocol_ssl/test.py::test_wrong_client_certificate 0.00s setup test_http_handlers_config/test.py::test_predefined_query_handler 0.00s setup test_hedged_requests/test.py::test_combination4 0.00s teardown test_hedged_requests/test.py::test_combination4 0.00s setup test_encrypted_disk_replication/test.py::test_zero_copy_replication 0.00s teardown test_distributed_load_balancing/test.py::test_distributed_replica_max_ignored_errors 0.00s teardown test_hedged_requests/test.py::test_send_table_status_sleep 0.00s teardown test_distributed_load_balancing/test.py::test_load_balancing_hostname_levenshtein_distance 0.00s teardown test_http_handlers_config/test.py::test_config_static_handler 0.00s setup test_distributed_format/test.py::test_two_files[test_cluster_internal_replication] 0.00s teardown test_grpc_protocol_ssl/test.py::test_insecure_channel 0.00s teardown test_distributed_inter_server_secret/test.py::test_secure_insert_async 0.00s teardown test_hedged_requests/test.py::test_receive_timeout2 0.00s setup test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_insecure_cluster[nopass-] 0.00s setup test_http_handlers_config/test.py::test_replicas_status_handler 0.00s setup test_endpoint_macro_substitution/test.py::test_select_by_type 0.00s setup test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[nopass-] 0.00s teardown test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_insecure_cluster[pass-foo] 0.00s teardown test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_secure_cluster[default-] 0.00s setup test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[pass-foo] 0.00s setup test_distributed_inter_server_secret/test.py::test_secure_cluster_distributed_over_distributed_different_users_cluster 0.00s teardown test_distributed_inter_server_secret/test.py::test_user_secure_cluster[default-] 0.00s setup test_hedged_requests/test.py::test_receive_timeout2 0.00s setup test_hedged_requests/test.py::test_stuck_replica 0.00s setup test_hedged_requests/test.py::test_send_table_status_sleep2 0.00s setup test_distributed_inter_server_secret/test.py::test_secure_insert_buffer_async 0.00s setup test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_secure_cluster[default-] 0.00s teardown test_distributed_format/test.py::test_single_file_old[test_cluster_no_internal_replication] 0.00s setup test_hedged_requests/test.py::test_combination3 0.00s setup test_distributed_inter_server_secret/test.py::test_insecure_insert_sync 0.00s setup test_grpc_protocol_ssl/test.py::test_secure_channel 0.00s teardown test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_insecure_cluster[nopass-] 0.00s teardown test_distributed_inter_server_secret/test.py::test_secure_insert_buffer_async 0.00s setup test_hedged_requests/test.py::test_send_data2 0.00s teardown test_hedged_requests/test.py::test_send_table_status_sleep2 0.00s setup test_http_handlers_config/test.py::test_config_static_handler 0.00s setup test_hedged_requests/test.py::test_send_table_status_sleep 0.00s teardown test_hedged_requests/test.py::test_combination2 0.00s teardown test_distributed_inter_server_secret/test.py::test_secure_cluster_distributed_over_distributed_different_users_cluster 0.00s teardown test_hedged_requests/test.py::test_long_query 0.00s teardown test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_insecure_cluster[default-] 0.00s setup test_distributed_inter_server_secret/test.py::test_secure_disagree 0.00s setup test_hedged_requests/test.py::test_send_data 0.00s setup test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_insecure_cluster[pass-foo] 0.00s teardown test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[default-] 0.00s setup test_http_handlers_config/test.py::test_prometheus_handler 0.00s teardown test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[default-] 0.00s teardown test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_insecure_cluster[pass-foo] 0.00s teardown test_distributed_inter_server_secret/test.py::test_secure 0.00s teardown test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[nopass-] 0.00s teardown test_distributed_inter_server_secret/test.py::test_insecure_insert_async 0.00s teardown test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[nopass-] 0.00s teardown test_distributed_inter_server_secret/test.py::test_secure_disagree 0.00s teardown test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_insecure_cluster[default-] =========================== short test summary info ============================ FAILED test_distributed_load_balancing/test.py::test_distributed_replica_max_ignored_errors FAILED test_hedged_requests/test.py::test_receive_timeout1 - Exception: Canno... FAILED test_hedged_requests/test.py::test_receive_timeout2 - Exception: Canno... FAILED test_hedged_requests/test.py::test_send_data - AssertionError: assert node_1 0 == node_2 0 FAILED test_hedged_requests/test.py::test_send_data2 - AssertionError: assert node_2 0 == node_3 0 PASSED test_grpc_protocol_ssl/test.py::test_insecure_channel PASSED test_grpc_protocol_ssl/test.py::test_secure_channel PASSED test_graphite_merge_tree/test.py::test_broken_partial_rollup PASSED test_grpc_protocol_ssl/test.py::test_wrong_client_certificate PASSED test_distributed_format/test.py::test_single_file[test_cluster_internal_replication] PASSED test_graphite_merge_tree/test.py::test_combined_rules PASSED test_distributed_format/test.py::test_single_file[test_cluster_no_internal_replication] PASSED test_http_handlers_config/test.py::test_absolute_path_static_handler PASSED test_graphite_merge_tree/test.py::test_combined_rules_with_default PASSED test_distributed_format/test.py::test_single_file_old[test_cluster_internal_replication] PASSED test_graphite_merge_tree/test.py::test_multiple_output_blocks PASSED test_graphite_merge_tree/test.py::test_multiple_paths_and_versions PASSED test_distributed_format/test.py::test_single_file_old[test_cluster_no_internal_replication] PASSED test_distributed_inter_server_secret/test.py::test_insecure PASSED test_distributed_format/test.py::test_two_files[test_cluster_internal_replication] PASSED test_distributed_inter_server_secret/test.py::test_insecure_insert_async PASSED test_distributed_format/test.py::test_two_files[test_cluster_no_internal_replication] PASSED test_distributed_inter_server_secret/test.py::test_insecure_insert_sync PASSED test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_insecure_cluster[default-] PASSED test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_insecure_cluster[nopass-] PASSED test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_insecure_cluster[pass-foo] PASSED test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_secure_cluster[default-] PASSED test_hedged_requests/test.py::test_async_connect PASSED test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_secure_cluster[nopass-] PASSED test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_secure_cluster[pass-foo] PASSED test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_insecure_cluster[default-] PASSED test_graphite_merge_tree/test.py::test_path_dangling_pointer PASSED test_graphite_merge_tree/test.py::test_paths_not_matching_any_pattern PASSED test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_insecure_cluster[nopass-] PASSED test_encrypted_disk_replication/test.py::test_replication PASSED test_graphite_merge_tree/test.py::test_rollup_aggregation PASSED test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_insecure_cluster[pass-foo] PASSED test_graphite_merge_tree/test.py::test_rollup_aggregation_2 PASSED test_encrypted_disk_replication/test.py::test_zero_copy_replication PASSED test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[default-] PASSED test_http_handlers_config/test.py::test_config_static_handler PASSED test_graphite_merge_tree/test.py::test_rollup_versions PASSED test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[nopass-] PASSED test_dotnet_client/test.py::test_dotnet_client PASSED test_graphite_merge_tree/test.py::test_system_graphite_retentions PASSED test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[pass-foo] PASSED test_graphite_merge_tree/test.py::test_wrong_rollup_config PASSED test_distributed_inter_server_secret/test.py::test_secure PASSED test_distributed_inter_server_secret/test.py::test_secure_cluster_distributed_over_distributed_different_users_cluster PASSED test_distributed_inter_server_secret/test.py::test_secure_cluster_distributed_over_distributed_different_users_remote PASSED test_distributed_inter_server_secret/test.py::test_secure_disagree PASSED test_distributed_inter_server_secret/test.py::test_secure_disagree_insert PASSED test_endpoint_macro_substitution/test.py::test_different_types PASSED test_distributed_inter_server_secret/test.py::test_secure_insert_async PASSED test_endpoint_macro_substitution/test.py::test_select_by_type PASSED test_hedged_requests/test.py::test_async_query_sending PASSED test_distributed_inter_server_secret/test.py::test_secure_insert_buffer_async PASSED test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_corrupted_big[0] PASSED test_distributed_inter_server_secret/test.py::test_secure_insert_sync PASSED test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[default-] PASSED test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[nopass-] PASSED test_http_handlers_config/test.py::test_defaults_http_handlers PASSED test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[pass-foo] PASSED test_hedged_requests/test.py::test_combination1 PASSED test_distributed_inter_server_secret/test.py::test_user_secure_cluster[default-] PASSED test_distributed_inter_server_secret/test.py::test_user_secure_cluster[nopass-] PASSED test_hot_reload_storage_policy/test.py::test_hot_reload_policy PASSED test_distributed_inter_server_secret/test.py::test_user_secure_cluster[pass-foo] PASSED test_executable_user_defined_functions_config_reload/test.py::test PASSED test_hedged_requests/test.py::test_combination2 PASSED test_hot_reload_storage_policy/test.py::test_hot_reload_policy_distributed_table PASSED test_distributed_insert_backward_compatibility/test.py::test_distributed_in_tuple PASSED test_hedged_requests/test.py::test_combination3 PASSED test_input_format_parallel_parsing_memory_tracking/test.py::test_memory_tracking_total PASSED test_http_handlers_config/test.py::test_defaults_http_handlers_config_order PASSED test_hedged_requests/test.py::test_combination4 PASSED test_distributed_load_balancing/test.py::test_load_balancing_default PASSED test_http_handlers_config/test.py::test_dynamic_query_handler PASSED test_hedged_requests/test.py::test_initial_receive_timeout PASSED test_http_handlers_config/test.py::test_fixed_static_handler PASSED test_hedged_requests/test.py::test_long_query PASSED test_http_handlers_config/test.py::test_headers_in_response PASSED test_globs_in_filepath/test.py::test_deep_structure PASSED test_globs_in_filepath/test.py::test_linear_structure PASSED test_http_handlers_config/test.py::test_predefined_query_handler PASSED test_globs_in_filepath/test.py::test_strange_filenames PASSED test_globs_in_filepath/test.py::test_table_function_and_virtual_columns PASSED test_http_handlers_config/test.py::test_prometheus_handler PASSED test_http_handlers_config/test.py::test_relative_path_static_handler PASSED test_http_handlers_config/test.py::test_replicas_status_handler PASSED test_distributed_load_balancing/test.py::test_load_balancing_first_or_random PASSED test_hedged_requests/test.py::test_send_table_status_sleep PASSED test_distributed_load_balancing/test.py::test_load_balancing_hostname_levenshtein_distance PASSED test_hedged_requests/test.py::test_send_table_status_sleep2 PASSED test_hedged_requests/test.py::test_stuck_replica PASSED test_distributed_load_balancing/test.py::test_load_balancing_in_order PASSED test_distributed_load_balancing/test.py::test_load_balancing_nearest_hostname PASSED test_distributed_load_balancing/test.py::test_load_balancing_priority_round_robin[dist_priority] PASSED test_distributed_load_balancing/test.py::test_load_balancing_priority_round_robin[dist_priority_negative] PASSED test_distributed_load_balancing/test.py::test_load_balancing_round_robin ================== 5 failed, 95 passed in 2079.21s (0:34:39) =================== Traceback (most recent call last): File "/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration/./runner", line 445, in subprocess.check_call(cmd, shell=True) File "/usr/lib/python3.10/subprocess.py", line 369, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command 'docker run --rm --name clickhouse_integration_tests_jvksbm --privileged --dns-search='.' --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-odbc-bridge:/clickhouse-odbc-bridge --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-library-bridge:/clickhouse-library-bridge --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/backupview:/ClickHouse/utils/backupview --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/grpc-client/pb2:/ClickHouse/utils/grpc-client/pb2 --volume=/run:/run/host:ro --volume=clickhouse_integration_tests_volume:/var/lib/docker -e DOCKER_DOTNET_CLIENT_TAG=11de0b29a15d -e DOCKER_HELPER_TAG=2cffe1eae894 -e DOCKER_BASE_TAG=1e0b53d756cf -e DOCKER_KERBEROS_KDC_TAG=9391ecdee8d7 -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=9bec2a638e6e -e DOCKER_MYSQL_JAVA_CLIENT_TAG=766bff31cfe4 -e DOCKER_MYSQL_JS_CLIENT_TAG=41ba7c2ec2a1 -e DOCKER_MYSQL_PHP_CLIENT_TAG=88be89c1e3b6 -e DOCKER_NGINX_DAV_TAG=b55ac9cd7519 -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=a4eff5c7f4d6 -e DOCKER_PYTHON_BOTTLE_TAG=caad4729259e -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600 -e CLICKHOUSE_USE_OLD_ANALYZER=1 -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS="--dist=loadfile -n 10 -rfEps --run-id=0 --color=no --durations=0 'test_distributed_format/test.py::test_single_file[test_cluster_internal_replication]' 'test_distributed_format/test.py::test_single_file[test_cluster_no_internal_replication]' 'test_distributed_format/test.py::test_single_file_old[test_cluster_internal_replication]' 'test_distributed_format/test.py::test_single_file_old[test_cluster_no_internal_replication]' 'test_distributed_format/test.py::test_two_files[test_cluster_internal_replication]' 'test_distributed_format/test.py::test_two_files[test_cluster_no_internal_replication]' test_distributed_insert_backward_compatibility/test.py::test_distributed_in_tuple test_distributed_inter_server_secret/test.py::test_insecure test_distributed_inter_server_secret/test.py::test_insecure_insert_async test_distributed_inter_server_secret/test.py::test_insecure_insert_sync 'test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_insecure_cluster[default-]' 'test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_insecure_cluster[nopass-]' 'test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_insecure_cluster[pass-foo]' 'test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_secure_cluster[default-]' 'test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_secure_cluster[nopass-]' 'test_distributed_inter_server_secret/test.py::test_per_user_inline_settings_secure_cluster[pass-foo]' 'test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_insecure_cluster[default-]' 'test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_insecure_cluster[nopass-]' 'test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_insecure_cluster[pass-foo]' 'test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[default-]' 'test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[nopass-]' 'test_distributed_inter_server_secret/test.py::test_per_user_protocol_settings_secure_cluster[pass-foo]' test_distributed_inter_server_secret/test.py::test_secure test_distributed_inter_server_secret/test.py::test_secure_cluster_distributed_over_distributed_different_users_cluster test_distributed_inter_server_secret/test.py::test_secure_cluster_distributed_over_distributed_different_users_remote test_distributed_inter_server_secret/test.py::test_secure_disagree test_distributed_inter_server_secret/test.py::test_secure_disagree_insert test_distributed_inter_server_secret/test.py::test_secure_insert_async test_distributed_inter_server_secret/test.py::test_secure_insert_buffer_async test_distributed_inter_server_secret/test.py::test_secure_insert_sync 'test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[default-]' 'test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[nopass-]' 'test_distributed_inter_server_secret/test.py::test_user_insecure_cluster[pass-foo]' 'test_distributed_inter_server_secret/test.py::test_user_secure_cluster[default-]' 'test_distributed_inter_server_secret/test.py::test_user_secure_cluster[nopass-]' 'test_distributed_inter_server_secret/test.py::test_user_secure_cluster[pass-foo]' test_distributed_load_balancing/test.py::test_distributed_replica_max_ignored_errors test_distributed_load_balancing/test.py::test_load_balancing_default test_distributed_load_balancing/test.py::test_load_balancing_first_or_random test_distributed_load_balancing/test.py::test_load_balancing_hostname_levenshtein_distance test_distributed_load_balancing/test.py::test_load_balancing_in_order test_distributed_load_balancing/test.py::test_load_balancing_nearest_hostname 'test_distributed_load_balancing/test.py::test_load_balancing_priority_round_robin[dist_priority]' 'test_distributed_load_balancing/test.py::test_load_balancing_priority_round_robin[dist_priority_negative]' test_distributed_load_balancing/test.py::test_load_balancing_round_robin test_dotnet_client/test.py::test_dotnet_client test_encrypted_disk_replication/test.py::test_replication test_encrypted_disk_replication/test.py::test_zero_copy_replication test_endpoint_macro_substitution/test.py::test_different_types test_endpoint_macro_substitution/test.py::test_select_by_type test_executable_user_defined_functions_config_reload/test.py::test test_globs_in_filepath/test.py::test_deep_structure test_globs_in_filepath/test.py::test_linear_structure test_globs_in_filepath/test.py::test_strange_filenames test_globs_in_filepath/test.py::test_table_function_and_virtual_columns test_graphite_merge_tree/test.py::test_broken_partial_rollup test_graphite_merge_tree/test.py::test_combined_rules test_graphite_merge_tree/test.py::test_combined_rules_with_default test_graphite_merge_tree/test.py::test_multiple_output_blocks test_graphite_merge_tree/test.py::test_multiple_paths_and_versions test_graphite_merge_tree/test.py::test_path_dangling_pointer test_graphite_merge_tree/test.py::test_paths_not_matching_any_pattern test_graphite_merge_tree/test.py::test_rollup_aggregation test_graphite_merge_tree/test.py::test_rollup_aggregation_2 test_graphite_merge_tree/test.py::test_rollup_versions test_graphite_merge_tree/test.py::test_system_graphite_retentions test_graphite_merge_tree/test.py::test_wrong_rollup_config test_grpc_protocol_ssl/test.py::test_insecure_channel test_grpc_protocol_ssl/test.py::test_secure_channel test_grpc_protocol_ssl/test.py::test_wrong_client_certificate test_hedged_requests/test.py::test_async_connect test_hedged_requests/test.py::test_async_query_sending test_hedged_requests/test.py::test_combination1 test_hedged_requests/test.py::test_combination2 test_hedged_requests/test.py::test_combination3 test_hedged_requests/test.py::test_combination4 test_hedged_requests/test.py::test_initial_receive_timeout test_hedged_requests/test.py::test_long_query test_hedged_requests/test.py::test_receive_timeout1 test_hedged_requests/test.py::test_receive_timeout2 test_hedged_requests/test.py::test_send_data test_hedged_requests/test.py::test_send_data2 test_hedged_requests/test.py::test_send_table_status_sleep test_hedged_requests/test.py::test_send_table_status_sleep2 test_hedged_requests/test.py::test_stuck_replica test_hot_reload_storage_policy/test.py::test_hot_reload_policy test_hot_reload_storage_policy/test.py::test_hot_reload_policy_distributed_table test_http_handlers_config/test.py::test_absolute_path_static_handler test_http_handlers_config/test.py::test_config_static_handler test_http_handlers_config/test.py::test_defaults_http_handlers test_http_handlers_config/test.py::test_defaults_http_handlers_config_order test_http_handlers_config/test.py::test_dynamic_query_handler test_http_handlers_config/test.py::test_fixed_static_handler test_http_handlers_config/test.py::test_headers_in_response test_http_handlers_config/test.py::test_predefined_query_handler test_http_handlers_config/test.py::test_prometheus_handler test_http_handlers_config/test.py::test_relative_path_static_handler test_http_handlers_config/test.py::test_replicas_status_handler test_input_format_parallel_parsing_memory_tracking/test.py::test_memory_tracking_total 'test_insert_distributed_async_send/test.py::test_insert_distributed_async_send_corrupted_big[0]' -vvv" altinityinfra/integration-tests-runner:37a9815fd2fa ' returned non-zero exit status 1.